 |
 |
I don't know the reason why there is a 16M limit and it clearly is not ideal for Ceph or HDFS type storage systems. Matevz, do you know? Can the limit be removed? Andy On Fri, 6 Sep 2019, Sam Skipsey wrote: > Hm. Adding that line to the config prevents the xrootd cache service from > staying up (and removing it makes it work again). > > 190906 19:03:37 19779 XrdFileCache_a2x: get block size 64M may not be > greater than 16777216 > 190906 19:03:37 19779 XrdFileCache_Manager: error Cache::Config() error in > parsing > > Setting it to the maximum, > > pfc.blocksize 16M > > does significantly improve the rate though (I get about 80 to 90% of the > direct rate - around 300MB/s). > > thanks. > > Is it possible to have the arbitrary limit on pfc.blocksize removed in > later releases? > > Sam > > > > On Fri, 6 Sep 2019 at 18:03, Yang, Wei < > [log in to unmask]> wrote: > >> Hi Sam, >> >> >> >> One thing that maybe useful for Xcache to talk to a CEPH storage is to set >> the Xcache "*pfc.blocksize*" to the same as the CEPH native bucket size. >> So if the CEPH bucket size is 64M, then you can add the following to >> cephc02's config >> >> >> >> pfc.blocksize 64M >> >> >> >> The default is 1M >> >> >> >> regards, >> >> -- >> >> Wei Yang | [log in to unmask] | 650-926-3338 (O) >> >> >> >> >> >> *From: *<[log in to unmask]> on behalf of Sam Skipsey < >> [log in to unmask]> >> *Date: *Friday, September 6, 2019 at 6:20 AM >> *To: *xrootd-l <[log in to unmask]> >> *Subject: *Help debugging slow Xrootd Proxy Cache <-> xrootd server >> transfers >> >> >> >> Hello everyone: >> >> >> >> I'm currently seeing some odd behaviour, and would appreciate some insight >> from people more deeply aware of xrootd than I am. >> >> >> >> Currently we are testing an xrootd setup using internal Xrootd disk >> caching proxies to improve performance against a Ceph object store. >> >> >> >> The configuration is: >> >> >> >> Xrootd server [with ceph plugin] on cephs02.beowulf.cluster >> >> >> >> Xrootd cache on cephc02.beowulf.cluster >> >> Cache is backed by a software raid-0 array of 6 SSDs. The SSD array >> achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck. >> >> Cephc02 is configured as a direct proxy for cephs02. >> >> >> >> Firewall is configured so all nodes are in the trusted zone relative to >> each other, so no ports blocked. >> >> >> >> The problem is that connections proxied through cephc02's xrootd server >> are extremely slow (10x to 15x slower) than direct connections to the >> xrootd server on cephs02. >> >> >> >> From cephc02, directly copying from cephs02: >> >> >> >> [root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB >> testfile2GB-in-1 >> >> [1.863GB/1.863GB][100%][==================================================][381.5MB/s] >> >> >> >> versus connecting via the proxy on cephc02 (cold cache): >> >> [root@cephc02 ~]# time xrdcp -v -v -v root:// >> 10.1.50.12:1094/ecpool:testfile2GB test-from-cache2 >> >> [1.863GB/1.863GB][100%][==================================================][20.51MB/s] >> >> >> >> (once the cache is warm, fetching from the cache itself is v fast, at > >> 1GB/s) >> >> >> >> >> >> Whilst I'd expect some caching overhead, this seems unuseably slow. >> >> What am I doing wrong here? >> >> >> >> Any help appreciated, >> >> >> >> Sam Skipsey >> >> University of Glasgow >> >> >> >> >> >> The Cache and Server authenticate with a shared secret, and their relevant >> configs are: >> >> >> >> cephc02: >> >> >> >> [root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg >> ofs.osslib libXrdPss.so >> pss.cachelib libXrdFileCache.so >> pfc.ram 16g >> pfc.trace info >> pfc.diskusage 0.90 0.95 >> oss.localroot /cache/ >> >> all.export /xroot:/ >> all.export /root:/ >> all.export * >> pss.origin 10.1.50.2:1095 >> xrd.allow host *.beowulf.cluster >> >> #inbound security protocol, for authenticating to the xrootd-ceph >> #sec.protocol >> xrootd.seclib /usr/lib64/libXrdSec.so >> sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c >> /etc/gateway/xrootd/sss.keytab.grp >> sec.protbind cephs02.beowulf.cluster:1095 only sss >> >> #outside security protocol, for authenticating users wanting to use the >> proxy >> #sec.protocol >> sec.protbind localhost only none >> >> xrd.report 127.0.0.1:9527 every 5s all >> >> >> >> - >> >> >> >> cephs02: >> >> >> >> # The export directive indicates which paths are to be exported. While the >> # default is '/tmp', we indicate it anyway to show you this directive. >> # >> all.export *? >> all.export / >> >> # The adminpath and pidpath variables indicate where the pid and various >> # IPC files should be placed >> # >> all.adminpath /var/spool/xrootd >> all.pidpath /var/run/xrootd >> xrootd.async segsize 67108864 >> xrd.buffers maxbsz 67108864 >> >> # Configure sss security - this is a shared secret between the xrootd-ceph >> and the xrootd-proxies, so the proxies are trusted to talk to the >> xrootd-ceph >> # >> xrootd.seclib /opt/xrootd/lib64/libXrdSec.so >> sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c >> /etc/gateway/xrootd/sss.keytab.grp >> sec.protbind * only sss >> >> xrootd.seclib /opt/xrootd/lib64/libXrdSec.so >> #sec.protocol host >> #sec.protbind localhost none >> >> # Configure rados connection >> # /this needs to be configured for the right stripe width >> ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool >> ,1,8388608,83886080 >> ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so >> xrootd.chksum adler32 >> >> # Configure the port >> # >> xrd.port 1095 >> >> >> >> >> ------------------------------ >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> >> ------------------------------ >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1