Hi Sam,
One thing that maybe useful for Xcache to talk to a CEPH storage is to set the Xcache "pfc.blocksize" to the same as the CEPH native bucket size. So if the CEPH bucket size is 64M, then you can add the following to cephc02's config
pfc.blocksize 64M
The default is 1M
regards,
--
Wei Yang | [log in to unmask] | 650-926-3338 (O)
From: <[log in to unmask]> on behalf of Sam Skipsey <[log in to unmask]>
Date: Friday, September 6, 2019 at 6:20 AM
To: xrootd-l <[log in to unmask]>
Subject: Help debugging slow Xrootd Proxy Cache <-> xrootd server transfers
Hello everyone:
I'm currently seeing some odd behaviour, and would appreciate some insight from people more deeply aware of xrootd than I am.
Currently we are testing an xrootd setup using internal Xrootd disk caching proxies to improve performance against a Ceph object store.
The configuration is:
Xrootd server [with ceph plugin] on cephs02.beowulf.cluster
Xrootd cache on cephc02.beowulf.cluster
Cache is backed by a software raid-0 array of 6 SSDs. The SSD array achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck.
Cephc02 is configured as a direct proxy for cephs02.
Firewall is configured so all nodes are in the trusted zone relative to each other, so no ports blocked.
The problem is that connections proxied through cephc02's xrootd server are extremely slow (10x to 15x slower) than direct connections to the xrootd server on cephs02.
From cephc02, directly copying from cephs02:
[root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB testfile2GB-in-1
[1.863GB/1.863GB][100%][==================================================][381.5MB/s]
versus connecting via the proxy on cephc02 (cold cache):
[root@cephc02 ~]# time xrdcp -v -v -v root://10.1.50.12:1094/ecpool:testfile2GB test-from-cache2
[1.863GB/1.863GB][100%][==================================================][20.51MB/s]
(once the cache is warm, fetching from the cache itself is v fast, at > 1GB/s)
Whilst I'd expect some caching overhead, this seems unuseably slow.
What am I doing wrong here?
Any help appreciated,
Sam Skipsey
University of Glasgow
The Cache and Server authenticate with a shared secret, and their relevant configs are:
cephc02:
[root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg
ofs.osslib libXrdPss.so
pss.cachelib libXrdFileCache.so
pfc.ram 16g
pfc.trace info
pfc.diskusage 0.90 0.95
oss.localroot /cache/
all.export /xroot:/
all.export /root:/
all.export *
pss.origin 10.1.50.2:1095
xrd.allow host *.beowulf.cluster
#inbound security protocol, for authenticating to the xrootd-ceph
#sec.protocol
xrootd.seclib /usr/lib64/libXrdSec.so
sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp
sec.protbind cephs02.beowulf.cluster:1095 only sss
#outside security protocol, for authenticating users wanting to use the proxy
#sec.protocol
sec.protbind localhost only none
xrd.report 127.0.0.1:9527 every 5s all
-
cephs02:
# The export directive indicates which paths are to be exported. While the
# default is '/tmp', we indicate it anyway to show you this directive.
#
all.export *?
all.export /
# The adminpath and pidpath variables indicate where the pid and various
# IPC files should be placed
#
all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
xrootd.async segsize 67108864
xrd.buffers maxbsz 67108864
# Configure sss security - this is a shared secret between the xrootd-ceph and the xrootd-proxies, so the proxies are trusted to talk to the xrootd-ceph
#
xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp
sec.protbind * only sss
xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
#sec.protocol host
#sec.protbind localhost none
# Configure rados connection
# /this needs to be configured for the right stripe width
ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool,1,8388608,83886080
ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so
xrootd.chksum adler32
# Configure the port
#
xrd.port 1095
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1