Print

Print

Hi Ewgenij,

Well, the confiug files are not exact'y clear on what the setup is.
Presumably you want a cluster of cache servers fronted by a single
redirector. It would seem that 192.168.1.186 is the redirectorbut that
192.168.1.103 is also decleared to be reditector. So, this setup won't do
what you want.

I assume yu have a cluster with a redirector, say "A" and a disk caching
proxy, say "B". The "A:1094" should be the origin and "B:1094" should
simply be an xrootd disk caching proxy. You would either talk to the
origin or the disk caching proxy but not both as the lines below indicate.

So, I would revise your config to accomplish that and your that should use
a single IP address in your tests. This is for a simple setup.

That said, I don't thnk (based on what you said) you want a forwarding
disk caching proxy.

Hope that helps.

Andrew

On Fri, 3 Dec 2021, eKatchko wrote:

> Hello,
>
> I hope this is the right place for this issue. If not, could you please direct me to a more suitable address for my issue?
>
> I am testing the xrootd network and the basics work fine but I am not able to list or locate data with xrdfs when using a clustered disc-caching-proxy in front of the xrootd data cluster.
> When I use
> ```
> xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ ls /
> ```
> it hangs and when I use
> ```
> xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ locate -r -d /testfile
> ```
> I get
> ```
> [ERROR] Server responded with an error: [3011] No servers have access to the file
> ```
> xrdcp on the other hand works fine and I am able to copy the file. Also using the xrdfs ls and locate commands work fine when talking directly to my data redirector.
>
> I am working on ubuntu and installed the packages by using this repository
> ```
> wget -O - https://storage-ci.web.cern.ch/storage-ci/storageci.key | sudo apt-key add -
> sudo add-apt-repository 'deb https://storage-ci.web.cern.ch/storage-ci/debian/xrootd bionic master'
> ```
>
> <details>
> <summary>My configuration</summary>
> Cache redirector
>
> ```
> all.manager 192.168.1.186:3121
> all.role manager
> cms.dfs lookup distrib mdhold 20m redirect immed
> if exec cmsd
> xrd.port 3121
> fi
> if exec xrootd
> xrd.port 1094
> fi
>
>
> cms.sched maxretries 0 nomultisrc space 98 fuzz 5 affinity weak
> cms.fxhold noloc 15m 4h
>
> set rootdir = /home/ubuntu/data
> all.export / cache
> oss.localroot $(rootdir)
> xrd.network keepalive kaparms 10m,1m,5
> xrd.timeout idle 60m
> all.adminpath /var/spool/xrootd
> all.pidpath /run/xrootd
> xrd.trace all -debug
> xrootd.trace all -debug
> ofs.trace all -debug
> cms.trace all -debug
> oss.trace all -debug
> ```
>
> Cache server:
>
> ```
> set rootdir = /home/ubuntu/data
> all.export / cache
> all.manager 192.168.1.186:3121
> oss.localroot $(rootdir)
> all.adminpath /var/spool/xrootd
> all.pidpath /run/xrootd
> cms.dfs lookup distrib mdhold 20m redirect immed
>
> if exec cmsd
> all.role server
> fi
> if exec xrootd
> all.role server
> set eBLOCKSIZE=1M
> set ePREFETCH=20
> set eRAMSIZE=2g
> set eSPACELOWWM=0.90
> set eSPACEHIGHWM=0.95
>
>
> pss.origin = xroot://192.168.1.103:1094
>
> ofs.osslib libXrdPss.so
> pss.cachelib libXrdFileCache.so
> xrd.network keepalive kaparms 10m,1m,5
> xrd.timeout idle 60m
> pfc.blocksize $eBLOCKSIZE
> pfc.ram $eRAMSIZE
> pfc.prefetch $ePREFETCH
> pfc.diskusage $eSPACELOWWM $eSPACEHIGHWM files 15g 16g 17g purgeinterval 60s
> fi
> xrd.trace all -debug
> xrootd.trace all -debug
> ofs.trace all -debug
> cms.trace all -debug
> oss.trace all -debug
> ```
>
> Data Redirector:
>
> ```
> all.export / r/o nostage
> all.manager 192.168.1.103:3121
> all.role manager
> cms.dfs lookup distrib mdhold 20m redirect immed
> set rootdir = /home/ubuntu/data
> oss.localroot $(rootdir)
> if exec xrootd
> xrd.port 1094
> fi
> if exec cmsd
> xrd.port 3121
> fi
> xrd.trace all -debug
> xrootd.trace all -debug
> ofs.trace all -debug
> cms.trace all -debug
> oss.trace all -debug
> all.adminpath /var/spool/xrootd
> all.pidpath /run/xrootd
> ```
>
> Data Server:
>
> ```
> all.export /
> set xrdr = 192.168.1.103
> all.manager $(xrdr):3121
> all.role server
> cms.space min 2g 5g
> all.adminpath /var/spool/xrootd
> all.pidpath /run/xrootd
> set rootdir = /vol
> oss.localroot $(rootdir)
> xrd.trace all -debug
> xrootd.trace all -debug
> ofs.trace all -debug
> cms.trace all -debug
> oss.trace all -debug
> ```
> </details>
>
> <details>
> <summary>Log output</summary>
> Cache redirector
>
> ```
> 211203 19:40:12 1200 Xrd_Sched: running underused thread monitor inq=0
> 211203 19:40:12 1200 Xrd_Sched: 3 threads; 1 idle
> 211203 19:40:12 1200 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:36 1300 Starting on Linux 4.15.0-163-generic
> 211203 19:40:36 1300 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
> Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
> Config warning: this hostname, cachered, is registered without a domain qualification.
> ++++++ xrootd ***@***.*** initialization started.
> Config using configuration file /etc/xrootd/xrootd-cache.cfg
> =====> xrd.port 1094
> =====> xrd.network keepalive kaparms 10m,1m,5
> =====> xrd.timeout idle 60m
> =====> all.adminpath /var/spool/xrootd
> =====> all.pidpath /run/xrootd
> =====> xrd.trace all -debug
> Config maximum number of connections restricted to 1048576
> 211203 19:40:36 1300 Xrd_Config: sendfile enabled.
> 211203 19:40:36 1300 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:36 1300 Xrd_Sched: Starting with 2 workers
> 211203 19:40:36 1300 Xrd_Sched: scheduling Idle link scan in 1200 seconds
> 211203 19:40:36 1300 Xrd_Poll: Starting poller 0
> 211203 19:40:36 1300 Xrd_Poll: Starting poller 1
> 211203 19:40:36 1300 Xrd_Poll: Starting poller 2
> 211203 19:40:36 1300 Xrd_Config: xroot:1094 wsz=131072
> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
> ++++++ xroot protocol initialization started.
> =====> all.export / cache
> =====> xrootd.trace all -debug
> Config exporting /
> Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
> Config Routing for 192.168.1.186: local pub4 prv4
> Config Route all4: 192.168.1.186 Dest=[::192.168.1.186]:1094
> ++++++ File system initialization started.
> =====> all.role manager
> =====> ofs.trace all -debug
> ++++++ Storage system initialization started.
> =====> all.export / cache
> =====> oss.localroot /home/ubuntu/data
> =====> oss.trace all -debug
> Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
> oss.alloc 0 0 0
> oss.spacescan 600
> oss.fdlimit 524288 1048576
> oss.maxsize 0
> oss.localroot /home/ubuntu/data
> oss.trace 7ff
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 2068011008
> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
> oss.path / r/o cache
> ------ Storage system initialization completed.
> ++++++ Configuring manager role. . .
> =====> all.manager 192.168.1.186:3121
> =====> all.adminpath /var/spool/xrootd
> =====> cms.trace all -debug
> Config 1 manager(s) started.
> Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
> all.role manager
> ofs.maxdelay 60
> ofs.persist manual hold 600 logdir /var/spool/xrootd/cache/.ofs/posc.log
> ofs.trace 7fff
> ------ File system manager initialization completed.
> Config asynchronous I/O has been disabled!
> 211203 19:40:36 1300 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
> 211203 19:40:36 1300 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
> 211203 19:40:36 1300 ofs_FAttr: FAttr req=info
> ------ xroot protocol initialization completed.
> 211203 19:40:36 1312 XrdOpen: Unable to connect socket to cachered; connection refused
> ------ xrootd ***@***.***:1094 initialization completed.
> 211203 19:40:36 1305 Xrd_Sched: running main accept inq=0
> 211203 19:40:40 1312 Xrd_Inet: Connected to cachered:3121
> 211203 19:40:40 1312 cms_ClientMan: Connected to suspended cachered v 3
> 211203 19:42:06 1312 cms_setStatus: Manager cachered resumed
> 211203 19:42:57 1304 Xrd_Sched: Now have 3 workers
> 211203 19:42:57 1304 Xrd_Sched: running main accept inq=0
> 211203 19:42:57 1305 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
> 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
> 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
> 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
> 211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
> 211203 19:42:57 1305 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/
> 211203 19:42:57 1312 cms_Decode: cachered redirects ubuntu.1219:7@[::ffff:192.168.1.149] to 192.168.1.54:1094 /
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-256 stat /
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 redirecting to 192.168.1.54:1094
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 16 data bytes; status=4004
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=2
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate nuD */
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_fsctl: locate args=*/
> 211203 19:42:57 1312 cms_Decode: cachered sent ubuntu.1219:7@[::ffff:192.168.1.149] 'Sr[::192.168.1.83]:1094 Sr[::192.168.1.54]:1094' */
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1024 locate */
> 211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 48 data bytes
> 211203 19:43:00 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
> 211203 19:43:00 1305 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149]
> 211203 19:43:39 1306 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup
> 211203 19:43:39 1344 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0
> 211203 19:43:39 1344 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42
> 211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7
> 211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0
> 211203 19:43:52 1304 Xrd_Inet: Accepted connection on port 1094 from 22@[::ffff:192.168.1.149]
> 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 attached to poller 0; num=1
> 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
> 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
> 211203 19:43:52 1305 Xrd_Sched: running main accept inq=0
> 211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
> 211203 19:43:52 1304 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] pvt IPv4 login
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate s /testfile
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile
> 211203 19:43:52 1312 cms_Decode: cachered delays ubuntu.1225:22@[::ffff:192.168.1.149] 5 /testfile
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=5 locate /testfile
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 stalling client for 5 sec
> 211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 4 data bytes; status=4005
> 211203 19:43:55 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
> 211203 19:43:55 1304 Xrd_Poll: Poller 0 enabled ubuntu.1225:22@[::ffff:192.168.1.149]
> 211203 19:43:57 1344 Xrd_Sched: running ubuntu.1225:22@[::ffff:192.168.1.149] inq=0
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate /testfile
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile
> 211203 19:43:57 1312 cms_Decode: cachered gave ubuntu.1225:22@[::ffff:192.168.1.149] err -2 'No servers have access to the file' /testfile
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1 locate /testfile
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending err 3011: No servers have access to the file
> 211203 19:43:57 1344 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] disc 0:00:05
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 22
> 211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 detached from poller 0; num=0
> ```
>
> Cache server one:
>
> ```
> 211203 19:40:37 1360 Starting on Linux 4.15.0-163-generic
> 211203 19:40:37 1360 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
> Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
> Config warning: this hostname, xcache-f1c1f, is registered without a domain qualification.
> ++++++ xrootd ***@***.*** initialization started.
> Config using configuration file /etc/xrootd/xrootd-cache.cfg
> =====> all.adminpath /var/spool/xrootd
> =====> all.pidpath /run/xrootd
> =====> xrd.network keepalive kaparms 10m,1m,5
> =====> xrd.timeout idle 60m
> =====> xrd.trace all -debug
> Config maximum number of connections restricted to 1048576
> 211203 19:40:37 1360 Xrd_Config: sendfile enabled.
> 211203 19:40:37 1360 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:37 1360 Xrd_Sched: Starting with 2 workers
> 211203 19:40:37 1360 Xrd_Sched: scheduling Idle link scan in 1200 seconds
> 211203 19:40:37 1360 Xrd_Poll: Starting poller 0
> 211203 19:40:37 1360 Xrd_Poll: Starting poller 1
> 211203 19:40:37 1360 Xrd_Poll: Starting poller 2
> 211203 19:40:37 1360 Xrd_Config: xroot:1094 wsz=131072
> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
> ++++++ xroot protocol initialization started.
> =====> all.export / cache
> =====> xrootd.trace all -debug
> Config exporting /
> Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
> Config Routing for 192.168.1.54: local pub4 prv4
> Config Route all4: 192.168.1.54 Dest=[::192.168.1.54]:1094
> ++++++ File system initialization started.
> =====> all.role server
> =====> ofs.osslib libXrdPss.so
> =====> ofs.trace all -debug
> Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67
> =====> all.export / cache
> =====> pss.origin = xroot://192.168.1.160:1094
> Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead!
> =====> pss.cachelib libXrdFileCache.so
> Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so
> ++++++ Proxy file cache initialization started.
> =====> pfc.blocksize 1M
> =====> pfc.ram 2g
> =====> pfc.prefetch 20
> =====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s
> ++++++ Storage system initialization started.
> =====> all.export / cache
> =====> oss.localroot /home/ubuntu/data
> =====> oss.trace all -debug
> ++++++ Configuring pfc mode . . .
> Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
> oss.alloc 0 0 0
> oss.spacescan 600
> oss.fdlimit 524288 1048576
> oss.maxsize 0
> oss.localroot /home/ubuntu/data
> oss.trace 7ff
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 2068011008
> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
> oss.path / r/o cache
> ------ Storage system initialization completed.
> Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration:
> pfc.cschk nocache net notls uvkeep lru
> pfc.blocksize 1048576
> pfc.prefetch 20
> pfc.ram 2g
> pfc.writequeue 16 4
> # Total available disk: 20629221376
> pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1
> pfc.spaces public public
> pfc.trace 2
> pfc.flush 2000
> pfc.acchistorysize 20
>
> Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive
> ------ Proxy File Cache configuration parsing completed
> ------ Proxy file cache initialization completed.
> ------ Proxy storage system initialization completed.
> ++++++ Configuring server role. . .
> =====> all.manager 192.168.1.186:3121
> =====> all.adminpath /var/spool/xrootd
> =====> cms.trace all -debug
> Config POSC has been disabled by the osslib plugin.
> Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
> all.role server
> ofs.maxdelay 60
> ofs.persist off hold 600
> ofs.trace 7fff
> ofs.osslib libXrdPss-5.so
> ------ File system server initialization completed.
> Config sendfile has been disabled by file system plugin.
> 211203 19:40:37 1360 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
> 211203 19:40:37 1360 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
> 211203 19:40:37 1360 ofs_FAttr: FAttr req=info
> ------ xroot protocol initialization completed.
> ------ xrootd ***@***.***:1094 initialization completed.
> 211203 19:40:37 1364 Xrd_Sched: running main accept inq=0
> 211203 19:40:37 1380 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused
> 211203 19:40:38 1379 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/)
> 211203 19:40:47 1380 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin
> 211203 19:42:57 1365 Xrd_Sched: Now have 3 workers
> 211203 19:42:57 1365 Xrd_Sched: running main accept inq=0
> 211203 19:42:57 1364 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
> 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
> 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
> 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
> 211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
> 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
> 211203 19:42:57 1364 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
> 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1
> 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/
> 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=0 stat /
> 211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 77 data bytes
> 211203 19:43:00 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
> 211203 19:43:00 1364 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149]
> 211203 19:43:39 1366 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup
> 211203 19:43:39 1407 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0
> 211203 19:43:39 1407 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42
> 211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7
> 211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0
> ```
>
> Cache server two:
>
> ```
> 211203 19:40:37 1447 Starting on Linux 4.15.0-163-generic
> 211203 19:40:37 1447 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
> Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
> Config warning: this hostname, xcache2-9e50f, is registered without a domain qualification.
> ++++++ xrootd ***@***.*** initialization started.
> Config using configuration file /etc/xrootd/xrootd-cache.cfg
> =====> all.adminpath /var/spool/xrootd
> =====> all.pidpath /run/xrootd
> =====> xrd.network keepalive kaparms 10m,1m,5
> =====> xrd.timeout idle 60m
> =====> xrd.trace all -debug
> Config maximum number of connections restricted to 1048576
> 211203 19:40:37 1447 Xrd_Config: sendfile enabled.
> 211203 19:40:37 1447 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:37 1447 Xrd_Sched: Starting with 2 workers
> 211203 19:40:37 1447 Xrd_Sched: scheduling Idle link scan in 1200 seconds
> 211203 19:40:37 1447 Xrd_Poll: Starting poller 0
> 211203 19:40:37 1447 Xrd_Poll: Starting poller 1
> 211203 19:40:37 1447 Xrd_Poll: Starting poller 2
> 211203 19:40:37 1447 Xrd_Config: xroot:1094 wsz=131072
> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
> ++++++ xroot protocol initialization started.
> =====> all.export / cache
> =====> xrootd.trace all -debug
> Config exporting /
> Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
> Config Routing for 192.168.1.83: local pub4 prv4
> Config Route all4: 192.168.1.83 Dest=[::192.168.1.83]:1094
> ++++++ File system initialization started.
> =====> all.role server
> =====> ofs.osslib libXrdPss.so
> =====> ofs.trace all -debug
> Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67
> =====> all.export / cache
> =====> pss.origin = xroot://192.168.1.160:1094
> Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead!
> =====> pss.cachelib libXrdFileCache.so
> Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so
> ++++++ Proxy file cache initialization started.
> =====> pfc.blocksize 1M
> =====> pfc.ram 2g
> =====> pfc.prefetch 20
> =====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s
> ++++++ Storage system initialization started.
> =====> all.export / cache
> =====> oss.localroot /home/ubuntu/data
> =====> oss.trace all -debug
> ++++++ Configuring pfc mode . . .
> Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
> oss.alloc 0 0 0
> oss.spacescan 600
> oss.fdlimit 524288 1048576
> oss.maxsize 0
> oss.localroot /home/ubuntu/data
> oss.trace 7ff
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 2068011008
> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
> oss.path / r/o cache
> ------ Storage system initialization completed.
> Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration:
> pfc.cschk nocache net notls uvkeep lru
> pfc.blocksize 1048576
> pfc.prefetch 20
> pfc.ram 2g
> pfc.writequeue 16 4
> # Total available disk: 20629221376
> pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1
> pfc.spaces public public
> pfc.trace 2
> pfc.flush 2000
> pfc.acchistorysize 20
>
> Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive
> ------ Proxy File Cache configuration parsing completed
> ------ Proxy file cache initialization completed.
> ------ Proxy storage system initialization completed.
> ++++++ Configuring server role. . .
> =====> all.manager 192.168.1.186:3121
> =====> all.adminpath /var/spool/xrootd
> =====> cms.trace all -debug
> Config POSC has been disabled by the osslib plugin.
> Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
> all.role server
> ofs.maxdelay 60
> ofs.persist off hold 600
> ofs.trace 7fff
> ofs.osslib libXrdPss-5.so
> ------ File system server initialization completed.
> Config sendfile has been disabled by file system plugin.
> 211203 19:40:38 1447 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
> 211203 19:40:38 1447 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
> 211203 19:40:38 1447 ofs_FAttr: FAttr req=info
> ------ xroot protocol initialization completed.
> ------ xrootd ***@***.***:1094 initialization completed.
> 211203 19:40:38 1451 Xrd_Sched: running main accept inq=0
> 211203 19:40:38 1467 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused
> 211203 19:40:38 1466 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/)
> 211203 19:40:48 1467 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin
> 211203 19:42:57 1452 Xrd_Sched: Now have 3 workers
> 211203 19:42:57 1452 Xrd_Sched: running main accept inq=0
> 211203 19:42:57 1451 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
> 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
> 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
> 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
> 211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
> 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
> 211203 19:42:57 1451 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
> 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=dirlist dlen=1
> 211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_opendir: fn=/
> ```
>
> Data redirector:
>
> ```
> 211203 19:40:44 1313 Starting on Linux 4.15.0-163-generic
> 211203 19:40:44 1313 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered
> Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
> Config warning: this hostname, xred-41f06, is registered without a domain qualification.
> ++++++ xrootd ***@***.*** initialization started.
> Config using configuration file /etc/xrootd/xrootd-clustered.cfg
> =====> xrd.port 1094
> =====> xrd.trace all -debug
> =====> all.adminpath /var/spool/xrootd
> =====> all.pidpath /run/xrootd
> Config maximum number of connections restricted to 1048576
> 211203 19:40:44 1313 Xrd_Config: sendfile enabled.
> 211203 19:40:44 1313 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:44 1313 Xrd_Sched: Starting with 2 workers
> 211203 19:40:44 1313 Xrd_Poll: Starting poller 0
> 211203 19:40:44 1313 Xrd_Poll: Starting poller 1
> 211203 19:40:44 1313 Xrd_Poll: Starting poller 2
> 211203 19:40:44 1313 Xrd_Config: xroot:1094 wsz=131072
> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
> ++++++ xroot protocol initialization started.
> =====> all.export / r/o
> =====> xrootd.trace all -debug
> Config exporting /
> Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
> Config Routing for 192.168.1.103: local pub4 prv4
> Config Route all4: 192.168.1.103 Dest=[::192.168.1.103]:1094
> ++++++ File system initialization started.
> =====> all.role manager
> =====> ofs.trace all -debug
> ++++++ Storage system initialization started.
> =====> all.export / r/o nostage
> =====> oss.localroot /home/ubuntu/data
> =====> oss.trace all -debug
> Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
> oss.alloc 0 0 0
> oss.spacescan 600
> oss.fdlimit 524288 1048576
> oss.maxsize 0
> oss.localroot /home/ubuntu/data
> oss.trace 7ff
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 2068015104
> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
> oss.path / r/o nocheck nodread nomig nopurge norcreate nostage
> ------ Storage system initialization completed.
> ++++++ Configuring manager role. . .
> =====> all.manager 192.168.1.103:3121
> =====> cms.trace all -debug
> =====> all.adminpath /var/spool/xrootd
> Config 1 manager(s) started.
> Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration:
> all.role manager
> ofs.maxdelay 60
> ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log
> ofs.trace 7fff
> ------ File system manager initialization completed.
> Config asynchronous I/O has been disabled!
> 211203 19:40:44 1313 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
> 211203 19:40:44 1313 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
> 211203 19:40:44 1313 ofs_FAttr: FAttr req=info
> ------ xroot protocol initialization completed.
> ------ xrootd ***@***.***:1094 initialization completed.
> 211203 19:40:44 1318 Xrd_Sched: running main accept inq=0
> 211203 19:40:44 1325 XrdOpen: Unable to connect socket to xred-41f06; connection refused
> 211203 19:40:48 1325 Xrd_Inet: Connected to xred-41f06:3121
> 211203 19:40:48 1325 cms_ClientMan: Connected to suspended xred-41f06 v 3
> 211203 19:42:14 1325 cms_setStatus: Manager xred-41f06 resumed
> ```
>
> Data server:
>
> ```
> 211203 19:40:45 1389 Starting on Linux 4.15.0-163-generic
> 211203 19:40:45 1389 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered
> Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
> Config warning: this hostname, xdata1-af575, is registered without a domain qualification.
> ++++++ xrootd ***@***.*** initialization started.
> Config using configuration file /etc/xrootd/xrootd-clustered.cfg
> =====> all.adminpath /var/spool/xrootd
> =====> all.pidpath /run/xrootd
> =====> xrd.trace all -debug
> Config maximum number of connections restricted to 1048576
> 211203 19:40:45 1389 Xrd_Config: sendfile enabled.
> 211203 19:40:45 1389 Xrd_Sched: scheduling underused thread monitor in 780 seconds
> 211203 19:40:45 1389 Xrd_Sched: Starting with 2 workers
> 211203 19:40:45 1389 Xrd_Poll: Starting poller 0
> 211203 19:40:45 1389 Xrd_Poll: Starting poller 1
> 211203 19:40:45 1389 Xrd_Poll: Starting poller 2
> 211203 19:40:45 1389 Xrd_Config: xroot:1094 wsz=131072
> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
> ++++++ xroot protocol initialization started.
> =====> all.export /
> =====> xrootd.trace all -debug
> Config exporting /
> Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
> Config Routing for 192.168.1.231: local pub4 prv4
> Config Route all4: 192.168.1.231 Dest=[::192.168.1.231]:1094
> ++++++ File system initialization started.
> =====> all.role server
> =====> ofs.trace all -debug
> ++++++ Storage system initialization started.
> =====> all.export /
> =====> oss.localroot /vol
> =====> oss.trace all -debug
> Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
> oss.alloc 0 0 0
> oss.spacescan 600
> oss.fdlimit 524288 1048576
> oss.maxsize 0
> oss.localroot /vol
> oss.trace 7ff
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 2068011008
> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
> oss.path / r/w nocheck nodread nomig nopurge norcreate nostage
> ------ Storage system initialization completed.
> ++++++ Configuring server role. . .
> =====> all.manager 192.168.1.103:3121
> =====> all.adminpath /var/spool/xrootd
> =====> cms.trace all -debug
> ++++++ Checkpoint initialization started.
> ++++++ Checkpoint initialization completed.
> Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration:
> all.role server
> ofs.maxdelay 60
> ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log
> ofs.trace 7fff
> ------ File system server initialization completed.
> Config asynchronous I/O has been disabled!
> 211203 19:40:45 1389 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
> 211203 19:40:45 1389 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
> 211203 19:40:45 1401 cms_Open: Unable to connect socket to /var/spool/xrootd/clustered/.olb/olbd.admin; connection refused
> 211203 19:40:45 1389 ofs_FAttr: FAttr req=info
> ------ xroot protocol initialization completed.
> ------ xrootd ***@***.***:1094 initialization completed.
> 211203 19:40:45 1394 Xrd_Sched: running main accept inq=0
> 211203 19:40:55 1401 cms_Finder: Connected to cmsd via /var/spool/xrootd/clustered/.olb/olbd.admin
> ```
> </details>
>
> Thanks in advance and kind regards,
> Ewgenij
>
> --
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/xrootd/xrootd/issues/1565

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/xrootd/xrootd/issues/1565#issuecomment-985915154", "url": "https://github.com/xrootd/xrootd/issues/1565#issuecomment-985915154", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1