I don't know the reason why there is a 16M limit and it clearly is not 
ideal for Ceph or HDFS type storage systems. Matevz, do you know? Can the 
limit be removed?

Andy

On Fri, 6 Sep 2019, Sam Skipsey wrote:

> Hm. Adding that line to the config prevents the xrootd cache service from
> staying up (and removing it makes it work again).
>
> 190906 19:03:37 19779 XrdFileCache_a2x: get block size 64M may not be
> greater than 16777216
> 190906 19:03:37 19779 XrdFileCache_Manager: error Cache::Config() error in
> parsing
>
> Setting it to the maximum,
>
> pfc.blocksize 16M
>
> does significantly improve the rate though (I get about 80 to 90% of the
> direct rate - around 300MB/s).
>
> thanks.
>
> Is it possible to have the arbitrary limit on pfc.blocksize removed in
> later releases?
>
> Sam
>
>
>
> On Fri, 6 Sep 2019 at 18:03, Yang, Wei <
> [log in to unmask]> wrote:
>
>> Hi Sam,
>>
>>
>>
>> One thing that maybe useful for Xcache to talk to a CEPH storage is to set
>> the Xcache "*pfc.blocksize*" to the same as the CEPH native bucket size.
>> So if the CEPH bucket size is 64M, then you can add the following to
>> cephc02's config
>>
>>
>>
>> pfc.blocksize 64M
>>
>>
>>
>> The default is 1M
>>
>>
>>
>> regards,
>>
>> --
>>
>> Wei Yang  |  [log in to unmask]  |  650-926-3338 (O)
>>
>>
>>
>>
>>
>> *From: *<[log in to unmask]> on behalf of Sam Skipsey <
>> [log in to unmask]>
>> *Date: *Friday, September 6, 2019 at 6:20 AM
>> *To: *xrootd-l <[log in to unmask]>
>> *Subject: *Help debugging slow Xrootd Proxy Cache <-> xrootd server
>> transfers
>>
>>
>>
>> Hello everyone:
>>
>>
>>
>> I'm currently seeing some odd behaviour, and would appreciate some insight
>> from people more deeply aware of xrootd than I am.
>>
>>
>>
>> Currently we are testing an xrootd setup using internal Xrootd disk
>> caching proxies to improve performance against a Ceph object store.
>>
>>
>>
>> The configuration is:
>>
>>
>>
>> Xrootd server [with ceph plugin] on cephs02.beowulf.cluster
>>
>>
>>
>> Xrootd cache on cephc02.beowulf.cluster
>>
>> Cache is backed by a software raid-0 array of 6 SSDs. The SSD array
>> achieves > 1.5GB/s transfer rates when tested, and is not an I/O bottleneck.
>>
>> Cephc02 is configured as a direct proxy for cephs02.
>>
>>
>>
>> Firewall is configured so all nodes are in the trusted zone relative to
>> each other, so no ports blocked.
>>
>>
>>
>> The problem is that connections proxied through cephc02's xrootd server
>> are extremely slow (10x to 15x slower) than direct connections to the
>> xrootd server on cephs02.
>>
>>
>>
>> From cephc02, directly copying from cephs02:
>>
>>
>>
>> [root@cephc02 ~]# time xrdcp root://cephs02:1095/ecpool:testfile2GB
>> testfile2GB-in-1
>>
>> [1.863GB/1.863GB][100%][==================================================][381.5MB/s]
>>
>>
>>
>> versus connecting via the proxy on cephc02 (cold cache):
>>
>> [root@cephc02 ~]# time xrdcp -v -v -v root://
>> 10.1.50.12:1094/ecpool:testfile2GB test-from-cache2
>>
>> [1.863GB/1.863GB][100%][==================================================][20.51MB/s]
>>
>>
>>
>> (once the cache is warm, fetching from the cache itself is v fast, at >
>> 1GB/s)
>>
>>
>>
>>
>>
>> Whilst I'd expect some caching overhead, this seems unuseably slow.
>>
>> What am I doing wrong here?
>>
>>
>>
>> Any help appreciated,
>>
>>
>>
>> Sam Skipsey
>>
>> University of Glasgow
>>
>>
>>
>>
>>
>> The Cache and Server authenticate with a shared secret, and their relevant
>> configs are:
>>
>>
>>
>> cephc02:
>>
>>
>>
>> [root@cephc02 ~]# cat /etc/xrootd/xrootd-cache.cfg
>> ofs.osslib    libXrdPss.so
>> pss.cachelib  libXrdFileCache.so
>> pfc.ram      16g
>> pfc.trace     info
>> pfc.diskusage 0.90 0.95
>> oss.localroot /cache/
>>
>> all.export /xroot:/
>> all.export /root:/
>> all.export *
>> pss.origin 10.1.50.2:1095
>> xrd.allow host *.beowulf.cluster
>>
>> #inbound security protocol, for authenticating to the xrootd-ceph
>> #sec.protocol
>> xrootd.seclib /usr/lib64/libXrdSec.so
>> sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c
>> /etc/gateway/xrootd/sss.keytab.grp
>> sec.protbind cephs02.beowulf.cluster:1095 only sss
>>
>> #outside security protocol, for authenticating users wanting to use the
>> proxy
>> #sec.protocol
>> sec.protbind localhost only none
>>
>> xrd.report 127.0.0.1:9527 every 5s all
>>
>>
>>
>> -
>>
>>
>>
>> cephs02:
>>
>>
>>
>> # The export directive indicates which paths are to be exported. While the
>> # default is '/tmp', we indicate it anyway to show you this directive.
>> #
>> all.export *?
>> all.export /
>>
>> # The adminpath and pidpath variables indicate where the pid and various
>> # IPC files should be placed
>> #
>> all.adminpath /var/spool/xrootd
>> all.pidpath /var/run/xrootd
>> xrootd.async segsize 67108864
>> xrd.buffers maxbsz 67108864
>>
>> # Configure sss security - this is a shared secret between the xrootd-ceph
>> and the xrootd-proxies, so the proxies are trusted to talk to the
>> xrootd-ceph
>> #
>> xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
>> sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c
>> /etc/gateway/xrootd/sss.keytab.grp
>> sec.protbind * only sss
>>
>> xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
>> #sec.protocol host
>> #sec.protbind localhost none
>>
>> # Configure rados connection
>> #   /this needs to be configured for the right stripe width
>> ofs.osslib +cksio /opt/xrootd/lib64/libXrdCeph.so admin@ecpool
>> ,1,8388608,83886080
>> ofs.xattrlib /opt/xrootd/lib64/libXrdCephXattr.so
>> xrootd.chksum adler32
>>
>> # Configure the port
>> #
>> xrd.port 1095
>>
>>
>>
>>
>> ------------------------------
>>
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>> ------------------------------
>>
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1