 |
 |
Hm. Adding that line to the config prevents the xrootd cache service from staying up (and removing it makes it work again). 190906 19:03:37 19779 XrdFileCache_a2x: get block size 64M may not be greater than 16777216 190906 19:03:37 19779 XrdFileCache_Manager: error Cache::Config() error in parsing Setting it to the maximum, pfc.blocksize 16M does significantly improve the rate though (I get about 80 to 90% of the direct rate - around 300MB/s). thanks. Is it possible to have the arbitrary limit on pfc.blocksize removed in later releases? Sam On Fri, 6 Sep 2019 at 18:03, Yang, Wei < [log in to unmask]> wrote: > Hi Sam, > > > > One thing that maybe useful for Xcache to talk to a CEPH storage is to set > the Xcache "*pfc.blocksize*" to the same as the CEPH native bucket size. > So if the CEPH bucket size is 64M, then you can add the following to > cephc02's config > > > > pfc.blocksize 64M > > > > The default is 1M > > > > regards, > > -- > > Wei Yang | [log in to unmask] | 650-926-3338 (O) > > > > > > *From: *<[log in to unmask]> on behalf of Sam Skipsey < > [log in to unmask]> > *Date: *Friday, September 6, 2019 at 6:20 AM > *To: *xrootd-l <[log in to unmask]> > *Subject: *Help debugging slow Xrootd Proxy Cache <-> xrootd server > transfers > > > > Hello everyone: > > > > I'm currently seeing some odd behaviour, and would appreciate some insight > from people more deeply aware of xrootd than I am. > > > > Currently we are testing an xrootd setup using internal Xrootd disk > caching proxies to improve performance against a Ceph object store. > > > > The configuration is: > > > > Xrootd server [with ceph plugin] on cephs02.beowulf.cluster > > > > Xrootd cache on cephc02.beowulf.cluster > > Cache is backed by a software raid-0 array of 6 SSDs. The SSD array > achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck. > > Cephc02 is configured as a direct proxy for cephs02. > > > > Firewall is configured so all nodes are in the trusted zone relative to > each other, so no ports blocked. > > > > The problem is that connections proxied through cephc02's xrootd server > are extremely slow (10x to 15x slower) than direct connections to the > xrootd server on cephs02. > > > > From cephc02, directly copying from cephs02: > > > > [root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB > testfile2GB-in-1 > > [1.863GB/1.863GB][100%][==================================================][381.5MB/s] > > > > versus connecting via the proxy on cephc02 (cold cache): > > [root@cephc02 ~]# time xrdcp -v -v -v root:// > 10.1.50.12:1094/ecpool:testfile2GB test-from-cache2 > > [1.863GB/1.863GB][100%][==================================================][20.51MB/s] > > > > (once the cache is warm, fetching from the cache itself is v fast, at > > 1GB/s) > > > > > > Whilst I'd expect some caching overhead, this seems unuseably slow. > > What am I doing wrong here? > > > > Any help appreciated, > > > > Sam Skipsey > > University of Glasgow > > > > > > The Cache and Server authenticate with a shared secret, and their relevant > configs are: > > > > cephc02: > > > > [root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg > ofs.osslib libXrdPss.so > pss.cachelib libXrdFileCache.so > pfc.ram 16g > pfc.trace info > pfc.diskusage 0.90 0.95 > oss.localroot /cache/ > > all.export /xroot:/ > all.export /root:/ > all.export * > pss.origin 10.1.50.2:1095 > xrd.allow host *.beowulf.cluster > > #inbound security protocol, for authenticating to the xrootd-ceph > #sec.protocol > xrootd.seclib /usr/lib64/libXrdSec.so > sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c > /etc/gateway/xrootd/sss.keytab.grp > sec.protbind cephs02.beowulf.cluster:1095 only sss > > #outside security protocol, for authenticating users wanting to use the > proxy > #sec.protocol > sec.protbind localhost only none > > xrd.report 127.0.0.1:9527 every 5s all > > > > - > > > > cephs02: > > > > # The export directive indicates which paths are to be exported. While the > # default is '/tmp', we indicate it anyway to show you this directive. > # > all.export *? > all.export / > > # The adminpath and pidpath variables indicate where the pid and various > # IPC files should be placed > # > all.adminpath /var/spool/xrootd > all.pidpath /var/run/xrootd > xrootd.async segsize 67108864 > xrd.buffers maxbsz 67108864 > > # Configure sss security - this is a shared secret between the xrootd-ceph > and the xrootd-proxies, so the proxies are trusted to talk to the > xrootd-ceph > # > xrootd.seclib /opt/xrootd/lib64/libXrdSec.so > sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c > /etc/gateway/xrootd/sss.keytab.grp > sec.protbind * only sss > > xrootd.seclib /opt/xrootd/lib64/libXrdSec.so > #sec.protocol host > #sec.protbind localhost none > > # Configure rados connection > # /this needs to be configured for the right stripe width > ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool > ,1,8388608,83886080 > ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so > xrootd.chksum adler32 > > # Configure the port > # > xrd.port 1095 > > > > > ------------------------------ > > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ------------------------------ > > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1