 |
 |
Hello everyone: I'm currently seeing some odd behaviour, and would appreciate some insight from people more deeply aware of xrootd than I am. Currently we are testing an xrootd setup using internal Xrootd disk caching proxies to improve performance against a Ceph object store. The configuration is: Xrootd server [with ceph plugin] on cephs02.beowulf.cluster Xrootd cache on cephc02.beowulf.cluster Cache is backed by a software raid-0 array of 6 SSDs. The SSD array achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck. Cephc02 is configured as a direct proxy for cephs02. Firewall is configured so all nodes are in the trusted zone relative to each other, so no ports blocked. The problem is that connections proxied through cephc02's xrootd server are extremely slow (10x to 15x slower) than direct connections to the xrootd server on cephs02. From cephc02, directly copying from cephs02: [root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB testfile2GB-in-1 [1.863GB/1.863GB][100%][==================================================][381.5MB/s] versus connecting via the proxy on cephc02 (cold cache): [root@cephc02 ~]# time xrdcp -v -v -v root:// 10.1.50.12:1094/ecpool:testfile2GB test-from-cache2 [1.863GB/1.863GB][100%][==================================================][20.51MB/s] (once the cache is warm, fetching from the cache itself is v fast, at > 1GB/s) Whilst I'd expect some caching overhead, this seems unuseably slow. What am I doing wrong here? Any help appreciated, Sam Skipsey University of Glasgow The Cache and Server authenticate with a shared secret, and their relevant configs are: cephc02: [root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg ofs.osslib libXrdPss.so pss.cachelib libXrdFileCache.so pfc.ram 16g pfc.trace info pfc.diskusage 0.90 0.95 oss.localroot /cache/ all.export /xroot:/ all.export /root:/ all.export * pss.origin 10.1.50.2:1095 xrd.allow host *.beowulf.cluster #inbound security protocol, for authenticating to the xrootd-ceph #sec.protocol xrootd.seclib /usr/lib64/libXrdSec.so sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp sec.protbind cephs02.beowulf.cluster:1095 only sss #outside security protocol, for authenticating users wanting to use the proxy #sec.protocol sec.protbind localhost only none xrd.report 127.0.0.1:9527 every 5s all - cephs02: # The export directive indicates which paths are to be exported. While the # default is '/tmp', we indicate it anyway to show you this directive. # all.export *? all.export / # The adminpath and pidpath variables indicate where the pid and various # IPC files should be placed # all.adminpath /var/spool/xrootd all.pidpath /var/run/xrootd xrootd.async segsize 67108864 xrd.buffers maxbsz 67108864 # Configure sss security - this is a shared secret between the xrootd-ceph and the xrootd-proxies, so the proxies are trusted to talk to the xrootd-ceph # xrootd.seclib /opt/xrootd/lib64/libXrdSec.so sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp sec.protbind * only sss xrootd.seclib /opt/xrootd/lib64/libXrdSec.so #sec.protocol host #sec.protbind localhost none # Configure rados connection # /this needs to be configured for the right stripe width ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool ,1,8388608,83886080 ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so xrootd.chksum adler32 # Configure the port # xrd.port 1095 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1