I'm currently seeing some odd behaviour, and would appreciate some insight from people more deeply aware of xrootd than I am.

Currently we are testing an xrootd setup using internal Xrootd disk caching proxies to improve performance against a Ceph object store.

The configuration is:

Xrootd server [with ceph plugin] on cephs02.beowulf.cluster

Xrootd cache on cephc02.beowulf.cluster

Cache is backed by a software raid-0 array of 6 SSDs. The SSD array achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck.

Cephc02 is configured as a direct proxy for cephs02.

Firewall is configured so all nodes are in the trusted zone relative to each other, so no ports blocked.

The problem is that connections proxied through cephc02's xrootd server are extremely slow (10x to 15x slower) than direct connections to the xrootd server on cephs02.

From cephc02, directly copying from cephs02:

[root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB testfile2GB-in-1
[1.863GB/1.863GB][100%][==================================================][381.5MB/s]

versus connecting via the proxy on cephc02 (cold cache):

[root@cephc02 ~]# time xrdcp -v -v -v root://10.1.50.12:1094/ecpool:testfile2GB test-from-cache2
[1.863GB/1.863GB][100%][==================================================][20.51MB/s]

(once the cache is warm, fetching from the cache itself is v fast, at > 1GB/s)

Whilst I'd expect some caching overhead, this seems unuseably slow. 

What am I doing wrong here?

Any help appreciated,

Sam Skipsey

University of Glasgow

The Cache and Server authenticate with a shared secret, and their relevant configs are:

cephc02:

[root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg
ofs.osslib    libXrdPss.so
pss.cachelib  libXrdFileCache.so
pfc.ram      16g
pfc.trace     info
pfc.diskusage 0.90 0.95
oss.localroot /cache/

all.export /xroot:/
all.export /root:/
all.export *
pss.origin 10.1.50.2:1095
xrd.allow host *.beowulf.cluster

#inbound security protocol, for authenticating to the xrootd-ceph
#sec.protocol
xrootd.seclib /usr/lib64/libXrdSec.so
sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp
sec.protbind cephs02.beowulf.cluster:1095 only sss

#outside security protocol, for authenticating users wanting to use the proxy
#sec.protocol
sec.protbind localhost only none

xrd.report 127.0.0.1:9527 every 5s all

-

cephs02:

# The export directive indicates which paths are to be exported. While the
# default is '/tmp', we indicate it anyway to show you this directive.
#
all.export *?
all.export /

# The adminpath and pidpath variables indicate where the pid and various
# IPC files should be placed
#
all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
xrootd.async segsize 67108864
xrd.buffers maxbsz 67108864

# Configure sss security - this is a shared secret between the xrootd-ceph and the xrootd-proxies, so the proxies are trusted to talk to the xrootd-ceph
#
xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c /etc/gateway/xrootd/sss.keytab.grp
sec.protbind * only sss

xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
#sec.protocol host
#sec.protbind localhost none

# Configure rados connection
#   /this needs to be configured for the right stripe width
ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool,1,8388608,83886080
ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so
xrootd.chksum adler32

# Configure the port
#
xrd.port 1095