 |
 |
Hi Ewgenij, Well, the confiug files are not exact'y clear on what the setup is. Presumably you want a cluster of cache servers fronted by a single redirector. It would seem that 192.168.1.186 is the redirectorbut that 192.168.1.103 is also decleared to be reditector. So, this setup won't do what you want. I assume yu have a cluster with a redirector, say "A" and a disk caching proxy, say "B". The "A:1094" should be the origin and "B:1094" should simply be an xrootd disk caching proxy. You would either talk to the origin or the disk caching proxy but not both as the lines below indicate. So, I would revise your config to accomplish that and your that should use a single IP address in your tests. This is for a simple setup. That said, I don't thnk (based on what you said) you want a forwarding disk caching proxy. Hope that helps. Andrew On Fri, 3 Dec 2021, eKatchko wrote: > Hello, > > I hope this is the right place for this issue. If not, could you please direct me to a more suitable address for my issue? > > I am testing the xrootd network and the basics work fine but I am not able to list or locate data with xrdfs when using a clustered disc-caching-proxy in front of the xrootd data cluster. > When I use > ``` > xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ ls / > ``` > it hangs and when I use > ``` > xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ locate -r -d /testfile > ``` > I get > ``` > [ERROR] Server responded with an error: [3011] No servers have access to the file > ``` > xrdcp on the other hand works fine and I am able to copy the file. Also using the xrdfs ls and locate commands work fine when talking directly to my data redirector. > > I am working on ubuntu and installed the packages by using this repository > ``` > wget -O - https://storage-ci.web.cern.ch/storage-ci/storageci.key | sudo apt-key add - > sudo add-apt-repository 'deb https://storage-ci.web.cern.ch/storage-ci/debian/xrootd bionic master' > ``` > > <details> > <summary>My configuration</summary> > Cache redirector > > ``` > all.manager 192.168.1.186:3121 > all.role manager > cms.dfs lookup distrib mdhold 20m redirect immed > if exec cmsd > xrd.port 3121 > fi > if exec xrootd > xrd.port 1094 > fi > > > cms.sched maxretries 0 nomultisrc space 98 fuzz 5 affinity weak > cms.fxhold noloc 15m 4h > > set rootdir = /home/ubuntu/data > all.export / cache > oss.localroot $(rootdir) > xrd.network keepalive kaparms 10m,1m,5 > xrd.timeout idle 60m > all.adminpath /var/spool/xrootd > all.pidpath /run/xrootd > xrd.trace all -debug > xrootd.trace all -debug > ofs.trace all -debug > cms.trace all -debug > oss.trace all -debug > ``` > > Cache server: > > ``` > set rootdir = /home/ubuntu/data > all.export / cache > all.manager 192.168.1.186:3121 > oss.localroot $(rootdir) > all.adminpath /var/spool/xrootd > all.pidpath /run/xrootd > cms.dfs lookup distrib mdhold 20m redirect immed > > if exec cmsd > all.role server > fi > if exec xrootd > all.role server > set eBLOCKSIZE=1M > set ePREFETCH=20 > set eRAMSIZE=2g > set eSPACELOWWM=0.90 > set eSPACEHIGHWM=0.95 > > > pss.origin = xroot://192.168.1.103:1094 > > ofs.osslib libXrdPss.so > pss.cachelib libXrdFileCache.so > xrd.network keepalive kaparms 10m,1m,5 > xrd.timeout idle 60m > pfc.blocksize $eBLOCKSIZE > pfc.ram $eRAMSIZE > pfc.prefetch $ePREFETCH > pfc.diskusage $eSPACELOWWM $eSPACEHIGHWM files 15g 16g 17g purgeinterval 60s > fi > xrd.trace all -debug > xrootd.trace all -debug > ofs.trace all -debug > cms.trace all -debug > oss.trace all -debug > ``` > > Data Redirector: > > ``` > all.export / r/o nostage > all.manager 192.168.1.103:3121 > all.role manager > cms.dfs lookup distrib mdhold 20m redirect immed > set rootdir = /home/ubuntu/data > oss.localroot $(rootdir) > if exec xrootd > xrd.port 1094 > fi > if exec cmsd > xrd.port 3121 > fi > xrd.trace all -debug > xrootd.trace all -debug > ofs.trace all -debug > cms.trace all -debug > oss.trace all -debug > all.adminpath /var/spool/xrootd > all.pidpath /run/xrootd > ``` > > Data Server: > > ``` > all.export / > set xrdr = 192.168.1.103 > all.manager $(xrdr):3121 > all.role server > cms.space min 2g 5g > all.adminpath /var/spool/xrootd > all.pidpath /run/xrootd > set rootdir = /vol > oss.localroot $(rootdir) > xrd.trace all -debug > xrootd.trace all -debug > ofs.trace all -debug > cms.trace all -debug > oss.trace all -debug > ``` > </details> > > <details> > <summary>Log output</summary> > Cache redirector > > ``` > 211203 19:40:12 1200 Xrd_Sched: running underused thread monitor inq=0 > 211203 19:40:12 1200 Xrd_Sched: 3 threads; 1 idle > 211203 19:40:12 1200 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:36 1300 Starting on Linux 4.15.0-163-generic > 211203 19:40:36 1300 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache > Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67 > Config warning: this hostname, cachered, is registered without a domain qualification. > ++++++ xrootd ***@***.*** initialization started. > Config using configuration file /etc/xrootd/xrootd-cache.cfg > =====> xrd.port 1094 > =====> xrd.network keepalive kaparms 10m,1m,5 > =====> xrd.timeout idle 60m > =====> all.adminpath /var/spool/xrootd > =====> all.pidpath /run/xrootd > =====> xrd.trace all -debug > Config maximum number of connections restricted to 1048576 > 211203 19:40:36 1300 Xrd_Config: sendfile enabled. > 211203 19:40:36 1300 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:36 1300 Xrd_Sched: Starting with 2 workers > 211203 19:40:36 1300 Xrd_Sched: scheduling Idle link scan in 1200 seconds > 211203 19:40:36 1300 Xrd_Poll: Starting poller 0 > 211203 19:40:36 1300 Xrd_Poll: Starting poller 1 > 211203 19:40:36 1300 Xrd_Poll: Starting poller 2 > 211203 19:40:36 1300 Xrd_Config: xroot:1094 wsz=131072 > Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67 > ++++++ xroot protocol initialization started. > =====> all.export / cache > =====> xrootd.trace all -debug > Config exporting / > Config warning: 'xrootd.seclib' not specified; strong authentication disabled! > Config Routing for 192.168.1.186: local pub4 prv4 > Config Route all4: 192.168.1.186 Dest=[::192.168.1.186]:1094 > ++++++ File system initialization started. > =====> all.role manager > =====> ofs.trace all -debug > ++++++ Storage system initialization started. > =====> all.export / cache > =====> oss.localroot /home/ubuntu/data > =====> oss.trace all -debug > Config effective /etc/xrootd/xrootd-cache.cfg oss configuration: > oss.alloc 0 0 0 > oss.spacescan 600 > oss.fdlimit 524288 1048576 > oss.maxsize 0 > oss.localroot /home/ubuntu/data > oss.trace 7ff > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 2068011008 > oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage > oss.path / r/o cache > ------ Storage system initialization completed. > ++++++ Configuring manager role. . . > =====> all.manager 192.168.1.186:3121 > =====> all.adminpath /var/spool/xrootd > =====> cms.trace all -debug > Config 1 manager(s) started. > Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration: > all.role manager > ofs.maxdelay 60 > ofs.persist manual hold 600 logdir /var/spool/xrootd/cache/.ofs/posc.log > ofs.trace 7fff > ------ File system manager initialization completed. > Config asynchronous I/O has been disabled! > 211203 19:40:36 1300 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds > 211203 19:40:36 1300 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds > 211203 19:40:36 1300 ofs_FAttr: FAttr req=info > ------ xroot protocol initialization completed. > 211203 19:40:36 1312 XrdOpen: Unable to connect socket to cachered; connection refused > ------ xrootd ***@***.***:1094 initialization completed. > 211203 19:40:36 1305 Xrd_Sched: running main accept inq=0 > 211203 19:40:40 1312 Xrd_Inet: Connected to cachered:3121 > 211203 19:40:40 1312 cms_ClientMan: Connected to suspended cachered v 3 > 211203 19:42:06 1312 cms_setStatus: Manager cachered resumed > 211203 19:42:57 1304 Xrd_Sched: Now have 3 workers > 211203 19:42:57 1304 Xrd_Sched: running main accept inq=0 > 211203 19:42:57 1305 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149] > 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1 > 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0 > 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0 > 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100 > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes > 211203 19:42:57 1305 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1 > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/ > 211203 19:42:57 1312 cms_Decode: cachered redirects ubuntu.1219:7@[::ffff:192.168.1.149] to 192.168.1.54:1094 / > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-256 stat / > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 redirecting to 192.168.1.54:1094 > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 16 data bytes; status=4004 > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=2 > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate nuD */ > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_fsctl: locate args=*/ > 211203 19:42:57 1312 cms_Decode: cachered sent ubuntu.1219:7@[::ffff:192.168.1.149] 'Sr[::192.168.1.83]:1094 Sr[::192.168.1.54]:1094' */ > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1024 locate */ > 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 48 data bytes > 211203 19:43:00 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes > 211203 19:43:00 1305 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149] > 211203 19:43:39 1306 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup > 211203 19:43:39 1344 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0 > 211203 19:43:39 1344 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42 > 211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7 > 211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0 > 211203 19:43:52 1304 Xrd_Inet: Accepted connection on port 1094 from 22@[::ffff:192.168.1.149] > 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 attached to poller 0; num=1 > 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0 > 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0 > 211203 19:43:52 1305 Xrd_Sched: running main accept inq=0 > 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100 > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes > 211203 19:43:52 1304 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] pvt IPv4 login > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9 > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate s /testfile > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile > 211203 19:43:52 1312 cms_Decode: cachered delays ubuntu.1225:22@[::ffff:192.168.1.149] 5 /testfile > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=5 locate /testfile > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 stalling client for 5 sec > 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 4 data bytes; status=4005 > 211203 19:43:55 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes > 211203 19:43:55 1304 Xrd_Poll: Poller 0 enabled ubuntu.1225:22@[::ffff:192.168.1.149] > 211203 19:43:57 1344 Xrd_Sched: running ubuntu.1225:22@[::ffff:192.168.1.149] inq=0 > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9 > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate /testfile > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile > 211203 19:43:57 1312 cms_Decode: cachered gave ubuntu.1225:22@[::ffff:192.168.1.149] err -2 'No servers have access to the file' /testfile > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1 locate /testfile > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending err 3011: No servers have access to the file > 211203 19:43:57 1344 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] disc 0:00:05 > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 22 > 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 detached from poller 0; num=0 > ``` > > Cache server one: > > ``` > 211203 19:40:37 1360 Starting on Linux 4.15.0-163-generic > 211203 19:40:37 1360 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache > Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67 > Config warning: this hostname, xcache-f1c1f, is registered without a domain qualification. > ++++++ xrootd ***@***.*** initialization started. > Config using configuration file /etc/xrootd/xrootd-cache.cfg > =====> all.adminpath /var/spool/xrootd > =====> all.pidpath /run/xrootd > =====> xrd.network keepalive kaparms 10m,1m,5 > =====> xrd.timeout idle 60m > =====> xrd.trace all -debug > Config maximum number of connections restricted to 1048576 > 211203 19:40:37 1360 Xrd_Config: sendfile enabled. > 211203 19:40:37 1360 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:37 1360 Xrd_Sched: Starting with 2 workers > 211203 19:40:37 1360 Xrd_Sched: scheduling Idle link scan in 1200 seconds > 211203 19:40:37 1360 Xrd_Poll: Starting poller 0 > 211203 19:40:37 1360 Xrd_Poll: Starting poller 1 > 211203 19:40:37 1360 Xrd_Poll: Starting poller 2 > 211203 19:40:37 1360 Xrd_Config: xroot:1094 wsz=131072 > Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67 > ++++++ xroot protocol initialization started. > =====> all.export / cache > =====> xrootd.trace all -debug > Config exporting / > Config warning: 'xrootd.seclib' not specified; strong authentication disabled! > Config Routing for 192.168.1.54: local pub4 prv4 > Config Route all4: 192.168.1.54 Dest=[::192.168.1.54]:1094 > ++++++ File system initialization started. > =====> all.role server > =====> ofs.osslib libXrdPss.so > =====> ofs.trace all -debug > Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67 > =====> all.export / cache > =====> pss.origin = xroot://192.168.1.160:1094 > Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead! > =====> pss.cachelib libXrdFileCache.so > Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so > ++++++ Proxy file cache initialization started. > =====> pfc.blocksize 1M > =====> pfc.ram 2g > =====> pfc.prefetch 20 > =====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s > ++++++ Storage system initialization started. > =====> all.export / cache > =====> oss.localroot /home/ubuntu/data > =====> oss.trace all -debug > ++++++ Configuring pfc mode . . . > Config effective /etc/xrootd/xrootd-cache.cfg oss configuration: > oss.alloc 0 0 0 > oss.spacescan 600 > oss.fdlimit 524288 1048576 > oss.maxsize 0 > oss.localroot /home/ubuntu/data > oss.trace 7ff > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 2068011008 > oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage > oss.path / r/o cache > ------ Storage system initialization completed. > Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration: > pfc.cschk nocache net notls uvkeep lru > pfc.blocksize 1048576 > pfc.prefetch 20 > pfc.ram 2g > pfc.writequeue 16 4 > # Total available disk: 20629221376 > pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1 > pfc.spaces public public > pfc.trace 2 > pfc.flush 2000 > pfc.acchistorysize 20 > > Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive > ------ Proxy File Cache configuration parsing completed > ------ Proxy file cache initialization completed. > ------ Proxy storage system initialization completed. > ++++++ Configuring server role. . . > =====> all.manager 192.168.1.186:3121 > =====> all.adminpath /var/spool/xrootd > =====> cms.trace all -debug > Config POSC has been disabled by the osslib plugin. > Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration: > all.role server > ofs.maxdelay 60 > ofs.persist off hold 600 > ofs.trace 7fff > ofs.osslib libXrdPss-5.so > ------ File system server initialization completed. > Config sendfile has been disabled by file system plugin. > 211203 19:40:37 1360 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds > 211203 19:40:37 1360 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds > 211203 19:40:37 1360 ofs_FAttr: FAttr req=info > ------ xroot protocol initialization completed. > ------ xrootd ***@***.***:1094 initialization completed. > 211203 19:40:37 1364 Xrd_Sched: running main accept inq=0 > 211203 19:40:37 1380 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused > 211203 19:40:38 1379 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/) > 211203 19:40:47 1380 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin > 211203 19:42:57 1365 Xrd_Sched: Now have 3 workers > 211203 19:42:57 1365 Xrd_Sched: running main accept inq=0 > 211203 19:42:57 1364 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149] > 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1 > 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0 > 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0 > 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100 > 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes > 211203 19:42:57 1364 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login > 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1 > 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/ > 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=0 stat / > 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 77 data bytes > 211203 19:43:00 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes > 211203 19:43:00 1364 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149] > 211203 19:43:39 1366 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup > 211203 19:43:39 1407 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0 > 211203 19:43:39 1407 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42 > 211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7 > 211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0 > ``` > > Cache server two: > > ``` > 211203 19:40:37 1447 Starting on Linux 4.15.0-163-generic > 211203 19:40:37 1447 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache > Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67 > Config warning: this hostname, xcache2-9e50f, is registered without a domain qualification. > ++++++ xrootd ***@***.*** initialization started. > Config using configuration file /etc/xrootd/xrootd-cache.cfg > =====> all.adminpath /var/spool/xrootd > =====> all.pidpath /run/xrootd > =====> xrd.network keepalive kaparms 10m,1m,5 > =====> xrd.timeout idle 60m > =====> xrd.trace all -debug > Config maximum number of connections restricted to 1048576 > 211203 19:40:37 1447 Xrd_Config: sendfile enabled. > 211203 19:40:37 1447 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:37 1447 Xrd_Sched: Starting with 2 workers > 211203 19:40:37 1447 Xrd_Sched: scheduling Idle link scan in 1200 seconds > 211203 19:40:37 1447 Xrd_Poll: Starting poller 0 > 211203 19:40:37 1447 Xrd_Poll: Starting poller 1 > 211203 19:40:37 1447 Xrd_Poll: Starting poller 2 > 211203 19:40:37 1447 Xrd_Config: xroot:1094 wsz=131072 > Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67 > ++++++ xroot protocol initialization started. > =====> all.export / cache > =====> xrootd.trace all -debug > Config exporting / > Config warning: 'xrootd.seclib' not specified; strong authentication disabled! > Config Routing for 192.168.1.83: local pub4 prv4 > Config Route all4: 192.168.1.83 Dest=[::192.168.1.83]:1094 > ++++++ File system initialization started. > =====> all.role server > =====> ofs.osslib libXrdPss.so > =====> ofs.trace all -debug > Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67 > =====> all.export / cache > =====> pss.origin = xroot://192.168.1.160:1094 > Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead! > =====> pss.cachelib libXrdFileCache.so > Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so > ++++++ Proxy file cache initialization started. > =====> pfc.blocksize 1M > =====> pfc.ram 2g > =====> pfc.prefetch 20 > =====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s > ++++++ Storage system initialization started. > =====> all.export / cache > =====> oss.localroot /home/ubuntu/data > =====> oss.trace all -debug > ++++++ Configuring pfc mode . . . > Config effective /etc/xrootd/xrootd-cache.cfg oss configuration: > oss.alloc 0 0 0 > oss.spacescan 600 > oss.fdlimit 524288 1048576 > oss.maxsize 0 > oss.localroot /home/ubuntu/data > oss.trace 7ff > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 2068011008 > oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage > oss.path / r/o cache > ------ Storage system initialization completed. > Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration: > pfc.cschk nocache net notls uvkeep lru > pfc.blocksize 1048576 > pfc.prefetch 20 > pfc.ram 2g > pfc.writequeue 16 4 > # Total available disk: 20629221376 > pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1 > pfc.spaces public public > pfc.trace 2 > pfc.flush 2000 > pfc.acchistorysize 20 > > Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive > ------ Proxy File Cache configuration parsing completed > ------ Proxy file cache initialization completed. > ------ Proxy storage system initialization completed. > ++++++ Configuring server role. . . > =====> all.manager 192.168.1.186:3121 > =====> all.adminpath /var/spool/xrootd > =====> cms.trace all -debug > Config POSC has been disabled by the osslib plugin. > Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration: > all.role server > ofs.maxdelay 60 > ofs.persist off hold 600 > ofs.trace 7fff > ofs.osslib libXrdPss-5.so > ------ File system server initialization completed. > Config sendfile has been disabled by file system plugin. > 211203 19:40:38 1447 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds > 211203 19:40:38 1447 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds > 211203 19:40:38 1447 ofs_FAttr: FAttr req=info > ------ xroot protocol initialization completed. > ------ xrootd ***@***.***:1094 initialization completed. > 211203 19:40:38 1451 Xrd_Sched: running main accept inq=0 > 211203 19:40:38 1467 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused > 211203 19:40:38 1466 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/) > 211203 19:40:48 1467 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin > 211203 19:42:57 1452 Xrd_Sched: Now have 3 workers > 211203 19:42:57 1452 Xrd_Sched: running main accept inq=0 > 211203 19:42:57 1451 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149] > 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1 > 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0 > 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0 > 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100 > 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes > 211203 19:42:57 1451 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login > 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=dirlist dlen=1 > 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_opendir: fn=/ > ``` > > Data redirector: > > ``` > 211203 19:40:44 1313 Starting on Linux 4.15.0-163-generic > 211203 19:40:44 1313 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered > Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67 > Config warning: this hostname, xred-41f06, is registered without a domain qualification. > ++++++ xrootd ***@***.*** initialization started. > Config using configuration file /etc/xrootd/xrootd-clustered.cfg > =====> xrd.port 1094 > =====> xrd.trace all -debug > =====> all.adminpath /var/spool/xrootd > =====> all.pidpath /run/xrootd > Config maximum number of connections restricted to 1048576 > 211203 19:40:44 1313 Xrd_Config: sendfile enabled. > 211203 19:40:44 1313 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:44 1313 Xrd_Sched: Starting with 2 workers > 211203 19:40:44 1313 Xrd_Poll: Starting poller 0 > 211203 19:40:44 1313 Xrd_Poll: Starting poller 1 > 211203 19:40:44 1313 Xrd_Poll: Starting poller 2 > 211203 19:40:44 1313 Xrd_Config: xroot:1094 wsz=131072 > Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67 > ++++++ xroot protocol initialization started. > =====> all.export / r/o > =====> xrootd.trace all -debug > Config exporting / > Config warning: 'xrootd.seclib' not specified; strong authentication disabled! > Config Routing for 192.168.1.103: local pub4 prv4 > Config Route all4: 192.168.1.103 Dest=[::192.168.1.103]:1094 > ++++++ File system initialization started. > =====> all.role manager > =====> ofs.trace all -debug > ++++++ Storage system initialization started. > =====> all.export / r/o nostage > =====> oss.localroot /home/ubuntu/data > =====> oss.trace all -debug > Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration: > oss.alloc 0 0 0 > oss.spacescan 600 > oss.fdlimit 524288 1048576 > oss.maxsize 0 > oss.localroot /home/ubuntu/data > oss.trace 7ff > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 2068015104 > oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage > oss.path / r/o nocheck nodread nomig nopurge norcreate nostage > ------ Storage system initialization completed. > ++++++ Configuring manager role. . . > =====> all.manager 192.168.1.103:3121 > =====> cms.trace all -debug > =====> all.adminpath /var/spool/xrootd > Config 1 manager(s) started. > Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration: > all.role manager > ofs.maxdelay 60 > ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log > ofs.trace 7fff > ------ File system manager initialization completed. > Config asynchronous I/O has been disabled! > 211203 19:40:44 1313 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds > 211203 19:40:44 1313 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds > 211203 19:40:44 1313 ofs_FAttr: FAttr req=info > ------ xroot protocol initialization completed. > ------ xrootd ***@***.***:1094 initialization completed. > 211203 19:40:44 1318 Xrd_Sched: running main accept inq=0 > 211203 19:40:44 1325 XrdOpen: Unable to connect socket to xred-41f06; connection refused > 211203 19:40:48 1325 Xrd_Inet: Connected to xred-41f06:3121 > 211203 19:40:48 1325 cms_ClientMan: Connected to suspended xred-41f06 v 3 > 211203 19:42:14 1325 cms_setStatus: Manager xred-41f06 resumed > ``` > > Data server: > > ``` > 211203 19:40:45 1389 Starting on Linux 4.15.0-163-generic > 211203 19:40:45 1389 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered > Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67 > Config warning: this hostname, xdata1-af575, is registered without a domain qualification. > ++++++ xrootd ***@***.*** initialization started. > Config using configuration file /etc/xrootd/xrootd-clustered.cfg > =====> all.adminpath /var/spool/xrootd > =====> all.pidpath /run/xrootd > =====> xrd.trace all -debug > Config maximum number of connections restricted to 1048576 > 211203 19:40:45 1389 Xrd_Config: sendfile enabled. > 211203 19:40:45 1389 Xrd_Sched: scheduling underused thread monitor in 780 seconds > 211203 19:40:45 1389 Xrd_Sched: Starting with 2 workers > 211203 19:40:45 1389 Xrd_Poll: Starting poller 0 > 211203 19:40:45 1389 Xrd_Poll: Starting poller 1 > 211203 19:40:45 1389 Xrd_Poll: Starting poller 2 > 211203 19:40:45 1389 Xrd_Config: xroot:1094 wsz=131072 > Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67 > ++++++ xroot protocol initialization started. > =====> all.export / > =====> xrootd.trace all -debug > Config exporting / > Config warning: 'xrootd.seclib' not specified; strong authentication disabled! > Config Routing for 192.168.1.231: local pub4 prv4 > Config Route all4: 192.168.1.231 Dest=[::192.168.1.231]:1094 > ++++++ File system initialization started. > =====> all.role server > =====> ofs.trace all -debug > ++++++ Storage system initialization started. > =====> all.export / > =====> oss.localroot /vol > =====> oss.trace all -debug > Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration: > oss.alloc 0 0 0 > oss.spacescan 600 > oss.fdlimit 524288 1048576 > oss.maxsize 0 > oss.localroot /vol > oss.trace 7ff > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 2068011008 > oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage > oss.path / r/w nocheck nodread nomig nopurge norcreate nostage > ------ Storage system initialization completed. > ++++++ Configuring server role. . . > =====> all.manager 192.168.1.103:3121 > =====> all.adminpath /var/spool/xrootd > =====> cms.trace all -debug > ++++++ Checkpoint initialization started. > ++++++ Checkpoint initialization completed. > Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration: > all.role server > ofs.maxdelay 60 > ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log > ofs.trace 7fff > ------ File system server initialization completed. > Config asynchronous I/O has been disabled! > 211203 19:40:45 1389 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds > 211203 19:40:45 1389 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds > 211203 19:40:45 1401 cms_Open: Unable to connect socket to /var/spool/xrootd/clustered/.olb/olbd.admin; connection refused > 211203 19:40:45 1389 ofs_FAttr: FAttr req=info > ------ xroot protocol initialization completed. > ------ xrootd ***@***.***:1094 initialization completed. > 211203 19:40:45 1394 Xrd_Sched: running main accept inq=0 > 211203 19:40:55 1401 cms_Finder: Connected to cmsd via /var/spool/xrootd/clustered/.olb/olbd.admin > ``` > </details> > > Thanks in advance and kind regards, > Ewgenij > > -- > You are receiving this because you are subscribed to this thread. > Reply to this email directly or view it on GitHub: > https://github.com/xrootd/xrootd/issues/1565 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/1565#issuecomment-985915154 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1