 |
 |
Hi Sam, I don't think the blocksize has to be 16M or less. It is probably an arbitrary number. Can you submit an issue request in xrootd github? http://www.github.com/xrootd/xrootd Xcache also do prefetching, thought I don't know whether this will help - depend on whether CEPH cache buckets for immediate use. regards, -- Wei Yang | [log in to unmask]<mailto:[log in to unmask]> | 650-926-3338 (O) From: <[log in to unmask]> on behalf of Sam Skipsey <[log in to unmask]> Date: Friday, September 6, 2019 at 11:16 AM To: "Yang, Wei" <[log in to unmask]> Cc: Sam Skipsey <[log in to unmask]>, xrootd-l <[log in to unmask]> Subject: Re: Help debugging slow Xrootd Proxy Cache <-> xrootd server transfers Hm. Adding that line to the config prevents the xrootd cache service from staying up (and removing it makes it work again). 190906 19:03:37 19779 XrdFileCache_a2x: get block size 64M may not be greater than 16777216 190906 19:03:37 19779 XrdFileCache_Manager: error Cache::Config() error in parsing Setting it to the maximum, pfc.blocksize 16M does significantly improve the rate though (I get about 80 to 90% of the direct rate - around 300MB/s). thanks. Is it possible to have the arbitrary limit on pfc.blocksize removed in later releases? Sam On Fri, 6 Sep 2019 at 18:03, Yang, Wei <[log in to unmask]<mailto:[log in to unmask]>> wrote: Hi Sam, One thing that maybe useful for Xcache to talk to a CEPH storage is to set the Xcache "pfc.blocksize" to the same as the CEPH native bucket size. So if the CEPH bucket size is 64M, then you can add the following to cephc02's config pfc.blocksize 64M The default is 1M regards, -- Wei Yang | [log in to unmask]<mailto:[log in to unmask]> | 650-926-3338 (O) From: <[log in to unmask]<mailto:[log in to unmask]>> on behalf of Sam Skipsey <[log in to unmask]<mailto:[log in to unmask]>> Date: Friday, September 6, 2019 at 6:20 AM To: xrootd-l <[log in to unmask]<mailto:[log in to unmask]>> Subject: Help debugging slow Xrootd Proxy Cache <-> xrootd server transfers Hello everyone: I'm currently seeing some odd behaviour, and would appreciate some insight from people more deeply aware of xrootd than I am. Currently we are testing an xrootd setup using internal Xrootd disk caching proxies to improve performance against a Ceph object store. The configuration is: Xrootd server [with ceph plugin] on cephs02.beowulf.cluster Xrootd cache on cephc02.beowulf.cluster Cache is backed by a software raid-0 array of 6 SSDs. The SSD array achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck. Cephc02 is configured as a direct proxy for cephs02. Firewall is configured so all nodes are in the trusted zone relative to each other, so no ports blocked. The problem is that connections proxied through cephc02's xrootd server are extremely slow (10x to 15x slower) than direct connections to the xrootd server on cephs02. From cephc02, directly copying from cephs02: [root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB testfile2GB-in-1 [1.863GB/1.863GB][100%][==================================================][381.5MB/s] versus connecting via the proxy on cephc02 (cold cache): [root@cephc02 ~]# time xrdcp -v -v -v root://10.1.50.12:1094/ecpool:testfile2GB<http://10.1.50.12:1094/ecpool:testfile2GB> test-from-cache2 [1.863GB/1.863GB][100%][==================================================][20.51MB/s] (once the cache is warm, fetching from the cache itself is v fast, at > 1GB/s) Whilst I'd expect some caching overhead, this seems unuseably slow. What am I doing wrong here? Any help appreciated, Sam Skipsey University of Glasgow The Cache and Server authenticate with a shared secret, and their relevant configs are: cephc02: [root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg ofs.osslib libXrdPss.so pss.cachelib libXrdFileCache.so pfc.ram 16g pfc.trace info pfc.diskusage 0.90 0.95 oss.localroot /cache/ all.export /xroot:/ all.export /root:/ all.export * pss.origin 10.1.50.2:1095<http://10.1.50.2:1095> xrd.allow host *.beowulf.cluster #inbound security protocol, for authenticating to the xrootd-ceph #sec.protocol xrootd.seclib /usr/lib64/libXrdSec.so sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp sec.protbind cephs02.beowulf.cluster:1095 only sss #outside security protocol, for authenticating users wanting to use the proxy #sec.protocol sec.protbind localhost only none xrd.report 127.0.0.1:9527<http://127.0.0.1:9527> every 5s all - cephs02: # The export directive indicates which paths are to be exported. While the # default is '/tmp', we indicate it anyway to show you this directive. # all.export *? all.export / # The adminpath and pidpath variables indicate where the pid and various # IPC files should be placed # all.adminpath /var/spool/xrootd all.pidpath /var/run/xrootd xrootd.async segsize 67108864 xrd.buffers maxbsz 67108864 # Configure sss security - this is a shared secret between the xrootd-ceph and the xrootd-proxies, so the proxies are trusted to talk to the xrootd-ceph # xrootd.seclib /opt/xrootd/lib64/libXrdSec.so sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp sec.protbind * only sss xrootd.seclib /opt/xrootd/lib64/libXrdSec.so #sec.protocol host #sec.protbind localhost none # Configure rados connection # /this needs to be configured for the right stripe width ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool,1,8388608,83886080 ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so xrootd.chksum adler32 # Configure the port # xrd.port 1095 ________________________________ Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ________________________________ Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ________________________________ Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1