Hello,
I hope this is the right place for this issue. If not, could you please direct me to a more suitable address for my issue?
I am testing the xrootd network and the basics work fine but I am not able to list or locate data with xrdfs when using a clustered disc-caching-proxy in front of the xrootd data cluster.
When I use
```
xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ ls /
```
it hangs and when I use
```
xrdfs xroot://192.168.1.186:1094//xroot://192.168.1.103:1094/ locate -r -d /testfile
```
I get
```
[ERROR] Server responded with an error: [3011] No servers have access to the file
```
xrdcp on the other hand works fine and I am able to copy the file. Also using the xrdfs ls and locate commands work fine when talking directly to my data redirector.
I am working on ubuntu and installed the packages by using this repository
```
wget -O - https://storage-ci.web.cern.ch/storage-ci/storageci.key | sudo apt-key add -
sudo add-apt-repository 'deb https://storage-ci.web.cern.ch/storage-ci/debian/xrootd bionic master'
```
<details>
<summary>My configuration</summary>
Cache redirector
```
all.manager 192.168.1.186:3121
all.role manager
cms.dfs lookup distrib mdhold 20m redirect immed
if exec cmsd
xrd.port 3121
fi
if exec xrootd
xrd.port 1094
fi
cms.sched maxretries 0 nomultisrc space 98 fuzz 5 affinity weak
cms.fxhold noloc 15m 4h
set rootdir = /home/ubuntu/data
all.export / cache
oss.localroot $(rootdir)
xrd.network keepalive kaparms 10m,1m,5
xrd.timeout idle 60m
all.adminpath /var/spool/xrootd
all.pidpath /run/xrootd
xrd.trace all -debug
xrootd.trace all -debug
ofs.trace all -debug
cms.trace all -debug
oss.trace all -debug
```
Cache server:
```
set rootdir = /home/ubuntu/data
all.export / cache
all.manager 192.168.1.186:3121
oss.localroot $(rootdir)
all.adminpath /var/spool/xrootd
all.pidpath /run/xrootd
cms.dfs lookup distrib mdhold 20m redirect immed
if exec cmsd
all.role server
fi
if exec xrootd
all.role server
set eBLOCKSIZE=1M
set ePREFETCH=20
set eRAMSIZE=2g
set eSPACELOWWM=0.90
set eSPACEHIGHWM=0.95
pss.origin = xroot://192.168.1.103:1094
ofs.osslib libXrdPss.so
pss.cachelib libXrdFileCache.so
xrd.network keepalive kaparms 10m,1m,5
xrd.timeout idle 60m
pfc.blocksize $eBLOCKSIZE
pfc.ram $eRAMSIZE
pfc.prefetch $ePREFETCH
pfc.diskusage $eSPACELOWWM $eSPACEHIGHWM files 15g 16g 17g purgeinterval 60s
fi
xrd.trace all -debug
xrootd.trace all -debug
ofs.trace all -debug
cms.trace all -debug
oss.trace all -debug
```
Data Redirector:
```
all.export / r/o nostage
all.manager 192.168.1.103:3121
all.role manager
cms.dfs lookup distrib mdhold 20m redirect immed
set rootdir = /home/ubuntu/data
oss.localroot $(rootdir)
if exec xrootd
xrd.port 1094
fi
if exec cmsd
xrd.port 3121
fi
xrd.trace all -debug
xrootd.trace all -debug
ofs.trace all -debug
cms.trace all -debug
oss.trace all -debug
all.adminpath /var/spool/xrootd
all.pidpath /run/xrootd
```
Data Server:
```
all.export /
set xrdr = 192.168.1.103
all.manager $(xrdr):3121
all.role server
cms.space min 2g 5g
all.adminpath /var/spool/xrootd
all.pidpath /run/xrootd
set rootdir = /vol
oss.localroot $(rootdir)
xrd.trace all -debug
xrootd.trace all -debug
ofs.trace all -debug
cms.trace all -debug
oss.trace all -debug
```
</details>
<details>
<summary>Log output</summary>
Cache redirector
```
211203 19:40:12 1200 Xrd_Sched: running underused thread monitor inq=0
211203 19:40:12 1200 Xrd_Sched: 3 threads; 1 idle
211203 19:40:12 1200 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:36 1300 Starting on Linux 4.15.0-163-generic
211203 19:40:36 1300 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
Config warning: this hostname, cachered, is registered without a domain qualification.
++++++ xrootd cache@cachered initialization started.
Config using configuration file /etc/xrootd/xrootd-cache.cfg
=====> xrd.port 1094
=====> xrd.network keepalive kaparms 10m,1m,5
=====> xrd.timeout idle 60m
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.trace all -debug
Config maximum number of connections restricted to 1048576
211203 19:40:36 1300 Xrd_Config: sendfile enabled.
211203 19:40:36 1300 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:36 1300 Xrd_Sched: Starting with 2 workers
211203 19:40:36 1300 Xrd_Sched: scheduling Idle link scan in 1200 seconds
211203 19:40:36 1300 Xrd_Poll: Starting poller 0
211203 19:40:36 1300 Xrd_Poll: Starting poller 1
211203 19:40:36 1300 Xrd_Poll: Starting poller 2
211203 19:40:36 1300 Xrd_Config: xroot:1094 wsz=131072
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
++++++ xroot protocol initialization started.
=====> all.export / cache
=====> xrootd.trace all -debug
Config exporting /
Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
Config Routing for 192.168.1.186: local pub4 prv4
Config Route all4: 192.168.1.186 Dest=[::192.168.1.186]:1094
++++++ File system initialization started.
=====> all.role manager
=====> ofs.trace all -debug
++++++ Storage system initialization started.
=====> all.export / cache
=====> oss.localroot /home/ubuntu/data
=====> oss.trace all -debug
Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.localroot /home/ubuntu/data
oss.trace 7ff
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 2068011008
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path / r/o cache
------ Storage system initialization completed.
++++++ Configuring manager role. . .
=====> all.manager 192.168.1.186:3121
=====> all.adminpath /var/spool/xrootd
=====> cms.trace all -debug
Config 1 manager(s) started.
Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
all.role manager
ofs.maxdelay 60
ofs.persist manual hold 600 logdir /var/spool/xrootd/cache/.ofs/posc.log
ofs.trace 7fff
------ File system manager initialization completed.
Config asynchronous I/O has been disabled!
211203 19:40:36 1300 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
211203 19:40:36 1300 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
211203 19:40:36 1300 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
211203 19:40:36 1312 XrdOpen: Unable to connect socket to cachered; connection refused
------ xrootd cache@cachered:1094 initialization completed.
211203 19:40:36 1305 Xrd_Sched: running main accept inq=0
211203 19:40:40 1312 Xrd_Inet: Connected to cachered:3121
211203 19:40:40 1312 cms_ClientMan: Connected to suspended cachered v 3
211203 19:42:06 1312 cms_setStatus: Manager cachered resumed
211203 19:42:57 1304 Xrd_Sched: Now have 3 workers
211203 19:42:57 1304 Xrd_Sched: running main accept inq=0
211203 19:42:57 1305 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
211203 19:42:57 1305 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
211203 19:42:57 1305 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/
211203 19:42:57 1312 cms_Decode: cachered redirects ubuntu.1219:7@[::ffff:192.168.1.149] to 192.168.1.54:1094 /
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-256 stat /
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 redirecting to 192.168.1.54:1094
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 16 data bytes; status=4004
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=2
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate nuD */
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_fsctl: locate args=*/
211203 19:42:57 1312 cms_Decode: cachered sent ubuntu.1219:7@[::ffff:192.168.1.149] 'Sr[::192.168.1.83]:1094 Sr[::192.168.1.54]:1094' */
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1024 locate */
211203 19:42:57 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 48 data bytes
211203 19:43:00 1305 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
211203 19:43:00 1305 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149]
211203 19:43:39 1306 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup
211203 19:43:39 1344 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0
211203 19:43:39 1344 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42
211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7
211203 19:43:39 1344 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0
211203 19:43:52 1304 Xrd_Inet: Accepted connection on port 1094 from 22@[::ffff:192.168.1.149]
211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 attached to poller 0; num=1
211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
211203 19:43:52 1305 Xrd_Sched: running main accept inq=0
211203 19:43:52 1304 anon.0:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
211203 19:43:52 1304 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] pvt IPv4 login
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate s /testfile
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile
211203 19:43:52 1312 cms_Decode: cachered delays ubuntu.1225:22@[::ffff:192.168.1.149] 5 /testfile
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=5 locate /testfile
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 stalling client for 5 sec
211203 19:43:52 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 4 data bytes; status=4005
211203 19:43:55 1304 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
211203 19:43:55 1304 Xrd_Poll: Poller 0 enabled ubuntu.1225:22@[::ffff:192.168.1.149]
211203 19:43:57 1344 Xrd_Sched: running ubuntu.1225:22@[::ffff:192.168.1.149] inq=0
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=locate dlen=9
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 locate /testfile
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] ofs_fsctl: locate args=/testfile
211203 19:43:57 1312 cms_Decode: cachered gave ubuntu.1225:22@[::ffff:192.168.1.149] err -2 'No servers have access to the file' /testfile
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=-1 locate /testfile
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending err 3011: No servers have access to the file
211203 19:43:57 1344 XrootdXeq: ubuntu.1225:22@[::ffff:192.168.1.149] disc 0:00:05
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 22
211203 19:43:57 1344 ubuntu.1225:22@[::ffff:192.168.1.149] Xrd_Poll: FD 22 detached from poller 0; num=0
```
Cache server one:
```
211203 19:40:37 1360 Starting on Linux 4.15.0-163-generic
211203 19:40:37 1360 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
Config warning: this hostname, xcache-f1c1f, is registered without a domain qualification.
++++++ xrootd cache@xcache-f1c1f initialization started.
Config using configuration file /etc/xrootd/xrootd-cache.cfg
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.network keepalive kaparms 10m,1m,5
=====> xrd.timeout idle 60m
=====> xrd.trace all -debug
Config maximum number of connections restricted to 1048576
211203 19:40:37 1360 Xrd_Config: sendfile enabled.
211203 19:40:37 1360 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:37 1360 Xrd_Sched: Starting with 2 workers
211203 19:40:37 1360 Xrd_Sched: scheduling Idle link scan in 1200 seconds
211203 19:40:37 1360 Xrd_Poll: Starting poller 0
211203 19:40:37 1360 Xrd_Poll: Starting poller 1
211203 19:40:37 1360 Xrd_Poll: Starting poller 2
211203 19:40:37 1360 Xrd_Config: xroot:1094 wsz=131072
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
++++++ xroot protocol initialization started.
=====> all.export / cache
=====> xrootd.trace all -debug
Config exporting /
Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
Config Routing for 192.168.1.54: local pub4 prv4
Config Route all4: 192.168.1.54 Dest=[::192.168.1.54]:1094
++++++ File system initialization started.
=====> all.role server
=====> ofs.osslib libXrdPss.so
=====> ofs.trace all -debug
Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67
=====> all.export / cache
=====> pss.origin = xroot://192.168.1.160:1094
Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead!
=====> pss.cachelib libXrdFileCache.so
Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so
++++++ Proxy file cache initialization started.
=====> pfc.blocksize 1M
=====> pfc.ram 2g
=====> pfc.prefetch 20
=====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s
++++++ Storage system initialization started.
=====> all.export / cache
=====> oss.localroot /home/ubuntu/data
=====> oss.trace all -debug
++++++ Configuring pfc mode . . .
Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.localroot /home/ubuntu/data
oss.trace 7ff
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 2068011008
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path / r/o cache
------ Storage system initialization completed.
Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration:
pfc.cschk nocache net notls uvkeep lru
pfc.blocksize 1048576
pfc.prefetch 20
pfc.ram 2g
pfc.writequeue 16 4
# Total available disk: 20629221376
pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1
pfc.spaces public public
pfc.trace 2
pfc.flush 2000
pfc.acchistorysize 20
Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive
------ Proxy File Cache configuration parsing completed
------ Proxy file cache initialization completed.
------ Proxy storage system initialization completed.
++++++ Configuring server role. . .
=====> all.manager 192.168.1.186:3121
=====> all.adminpath /var/spool/xrootd
=====> cms.trace all -debug
Config POSC has been disabled by the osslib plugin.
Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
all.role server
ofs.maxdelay 60
ofs.persist off hold 600
ofs.trace 7fff
ofs.osslib libXrdPss-5.so
------ File system server initialization completed.
Config sendfile has been disabled by file system plugin.
211203 19:40:37 1360 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
211203 19:40:37 1360 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
211203 19:40:37 1360 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
------ xrootd cache@xcache-f1c1f:1094 initialization completed.
211203 19:40:37 1364 Xrd_Sched: running main accept inq=0
211203 19:40:37 1380 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused
211203 19:40:38 1379 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/)
211203 19:40:47 1380 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin
211203 19:42:57 1365 Xrd_Sched: Now have 3 workers
211203 19:42:57 1365 Xrd_Sched: running main accept inq=0
211203 19:42:57 1364 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
211203 19:42:57 1364 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
211203 19:42:57 1364 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=stat dlen=1
211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_stat: fn=/
211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 rc=0 stat /
211203 19:42:57 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0100 sending 77 data bytes
211203 19:43:00 1364 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 request timeout; read 0 of 24 bytes
211203 19:43:00 1364 Xrd_Poll: Poller 0 enabled ubuntu.1219:7@[::ffff:192.168.1.149]
211203 19:43:39 1366 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Link 7 terminating: hangup
211203 19:43:39 1407 Xrd_Sched: running ubuntu.1219:7@[::ffff:192.168.1.149] inq=0
211203 19:43:39 1407 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] disc 0:00:42
211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: Poller 0 removing FD 7
211203 19:43:39 1407 ubuntu.1219:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 detached from poller 0; num=0
```
Cache server two:
```
211203 19:40:37 1447 Starting on Linux 4.15.0-163-generic
211203 19:40:37 1447 /usr/bin/xrootd -b -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-cache.cfg -k fifo -s /run/xrootd/xrootd-cache.pid -n cache
Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
Config warning: this hostname, xcache2-9e50f, is registered without a domain qualification.
++++++ xrootd cache@xcache2-9e50f initialization started.
Config using configuration file /etc/xrootd/xrootd-cache.cfg
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.network keepalive kaparms 10m,1m,5
=====> xrd.timeout idle 60m
=====> xrd.trace all -debug
Config maximum number of connections restricted to 1048576
211203 19:40:37 1447 Xrd_Config: sendfile enabled.
211203 19:40:37 1447 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:37 1447 Xrd_Sched: Starting with 2 workers
211203 19:40:37 1447 Xrd_Sched: scheduling Idle link scan in 1200 seconds
211203 19:40:37 1447 Xrd_Poll: Starting poller 0
211203 19:40:37 1447 Xrd_Poll: Starting poller 1
211203 19:40:37 1447 Xrd_Poll: Starting poller 2
211203 19:40:37 1447 Xrd_Config: xroot:1094 wsz=131072
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
++++++ xroot protocol initialization started.
=====> all.export / cache
=====> xrootd.trace all -debug
Config exporting /
Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
Config Routing for 192.168.1.83: local pub4 prv4
Config Route all4: 192.168.1.83 Dest=[::192.168.1.83]:1094
++++++ File system initialization started.
=====> all.role server
=====> ofs.osslib libXrdPss.so
=====> ofs.trace all -debug
Copr. 2019, Stanford University, Pss Version v20211203-ffc4c6b67
=====> all.export / cache
=====> pss.origin = xroot://192.168.1.160:1094
Config warning: 'libXrdFileCache' has been replaced by 'libXrdPfc'; for future compatibility specify 'default' instead!
=====> pss.cachelib libXrdFileCache.so
Plugin loaded unreleased XrdPfc v20211203-ffc4c6b67 from cachelib libXrdPfc-5.so
++++++ Proxy file cache initialization started.
=====> pfc.blocksize 1M
=====> pfc.ram 2g
=====> pfc.prefetch 20
=====> pfc.diskusage 0.90 0.95 files 15g 16g 17g purgeinterval 60s
++++++ Storage system initialization started.
=====> all.export / cache
=====> oss.localroot /home/ubuntu/data
=====> oss.trace all -debug
++++++ Configuring pfc mode . . .
Config effective /etc/xrootd/xrootd-cache.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.localroot /home/ubuntu/data
oss.trace 7ff
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 2068011008
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path / r/o cache
------ Storage system initialization completed.
Config effective /etc/xrootd/xrootd-cache.cfg pfc configuration:
pfc.cschk nocache net notls uvkeep lru
pfc.blocksize 1048576
pfc.prefetch 20
pfc.ram 2g
pfc.writequeue 16 4
# Total available disk: 20629221376
pfc.diskusage 18566299238 19597760307 files 16106127360 17179869184 18253611008 purgeinterval 60 purgecoldfiles -1
pfc.spaces public public
pfc.trace 2
pfc.flush 2000
pfc.acchistorysize 20
Config Proxy File Cache g-stream has NOT been configured via xrootd.monitor directive
------ Proxy File Cache configuration parsing completed
------ Proxy file cache initialization completed.
------ Proxy storage system initialization completed.
++++++ Configuring server role. . .
=====> all.manager 192.168.1.186:3121
=====> all.adminpath /var/spool/xrootd
=====> cms.trace all -debug
Config POSC has been disabled by the osslib plugin.
Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
all.role server
ofs.maxdelay 60
ofs.persist off hold 600
ofs.trace 7fff
ofs.osslib libXrdPss-5.so
------ File system server initialization completed.
Config sendfile has been disabled by file system plugin.
211203 19:40:38 1447 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
211203 19:40:38 1447 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
211203 19:40:38 1447 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
------ xrootd cache@xcache2-9e50f:1094 initialization completed.
211203 19:40:38 1451 Xrd_Sched: running main accept inq=0
211203 19:40:38 1467 cms_Open: Unable to connect socket to /var/spool/xrootd/cache/.olb/olbd.admin; connection refused
211203 19:40:38 1466 ubuntu oss_Opendir: lcl path /home/ubuntu/data/ (/)
211203 19:40:48 1467 cms_Finder: Connected to cmsd via /var/spool/xrootd/cache/.olb/olbd.admin
211203 19:42:57 1452 Xrd_Sched: Now have 3 workers
211203 19:42:57 1452 Xrd_Sched: running main accept inq=0
211203 19:42:57 1451 Xrd_Inet: Accepted connection on port 1094 from 7@[::ffff:192.168.1.149]
211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrd_Poll: FD 7 attached to poller 0; num=1
211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=protocol dlen=0
211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 8 data bytes; status=0
211203 19:42:57 1451 anon.0:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0000 req=login dlen=100
211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Response: 0000 sending 16 data bytes
211203 19:42:57 1451 XrootdXeq: ubuntu.1219:7@[::ffff:192.168.1.149] pvt IPv4 login
211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] Xrootd_Protocol: 0100 req=dirlist dlen=1
211203 19:42:57 1451 ubuntu.1219:7@[::ffff:192.168.1.149] ofs_opendir: fn=/
```
Data redirector:
```
211203 19:40:44 1313 Starting on Linux 4.15.0-163-generic
211203 19:40:44 1313 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered
Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
Config warning: this hostname, xred-41f06, is registered without a domain qualification.
++++++ xrootd clustered@xred-41f06 initialization started.
Config using configuration file /etc/xrootd/xrootd-clustered.cfg
=====> xrd.port 1094
=====> xrd.trace all -debug
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
Config maximum number of connections restricted to 1048576
211203 19:40:44 1313 Xrd_Config: sendfile enabled.
211203 19:40:44 1313 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:44 1313 Xrd_Sched: Starting with 2 workers
211203 19:40:44 1313 Xrd_Poll: Starting poller 0
211203 19:40:44 1313 Xrd_Poll: Starting poller 1
211203 19:40:44 1313 Xrd_Poll: Starting poller 2
211203 19:40:44 1313 Xrd_Config: xroot:1094 wsz=131072
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
++++++ xroot protocol initialization started.
=====> all.export / r/o
=====> xrootd.trace all -debug
Config exporting /
Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
Config Routing for 192.168.1.103: local pub4 prv4
Config Route all4: 192.168.1.103 Dest=[::192.168.1.103]:1094
++++++ File system initialization started.
=====> all.role manager
=====> ofs.trace all -debug
++++++ Storage system initialization started.
=====> all.export / r/o nostage
=====> oss.localroot /home/ubuntu/data
=====> oss.trace all -debug
Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.localroot /home/ubuntu/data
oss.trace 7ff
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 2068015104
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path / r/o nocheck nodread nomig nopurge norcreate nostage
------ Storage system initialization completed.
++++++ Configuring manager role. . .
=====> all.manager 192.168.1.103:3121
=====> cms.trace all -debug
=====> all.adminpath /var/spool/xrootd
Config 1 manager(s) started.
Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration:
all.role manager
ofs.maxdelay 60
ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log
ofs.trace 7fff
------ File system manager initialization completed.
Config asynchronous I/O has been disabled!
211203 19:40:44 1313 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
211203 19:40:44 1313 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
211203 19:40:44 1313 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
------ xrootd clustered@xred-41f06:1094 initialization completed.
211203 19:40:44 1318 Xrd_Sched: running main accept inq=0
211203 19:40:44 1325 XrdOpen: Unable to connect socket to xred-41f06; connection refused
211203 19:40:48 1325 Xrd_Inet: Connected to xred-41f06:3121
211203 19:40:48 1325 cms_ClientMan: Connected to suspended xred-41f06 v 3
211203 19:42:14 1325 cms_setStatus: Manager xred-41f06 resumed
```
Data server:
```
211203 19:40:45 1389 Starting on Linux 4.15.0-163-generic
211203 19:40:45 1389 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -b -c /etc/xrootd/xrootd-clustered.cfg -k fifo -s /run/xrootd/xrootd-clustered.pid -n clustered
Copr. 2004-2012 Stanford University, xrd version v20211203-ffc4c6b67
Config warning: this hostname, xdata1-af575, is registered without a domain qualification.
++++++ xrootd clustered@xdata1-af575 initialization started.
Config using configuration file /etc/xrootd/xrootd-clustered.cfg
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.trace all -debug
Config maximum number of connections restricted to 1048576
211203 19:40:45 1389 Xrd_Config: sendfile enabled.
211203 19:40:45 1389 Xrd_Sched: scheduling underused thread monitor in 780 seconds
211203 19:40:45 1389 Xrd_Sched: Starting with 2 workers
211203 19:40:45 1389 Xrd_Poll: Starting poller 0
211203 19:40:45 1389 Xrd_Poll: Starting poller 1
211203 19:40:45 1389 Xrd_Poll: Starting poller 2
211203 19:40:45 1389 Xrd_Config: xroot:1094 wsz=131072
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v20211203-ffc4c6b67
++++++ xroot protocol initialization started.
=====> all.export /
=====> xrootd.trace all -debug
Config exporting /
Config warning: 'xrootd.seclib' not specified; strong authentication disabled!
Config Routing for 192.168.1.231: local pub4 prv4
Config Route all4: 192.168.1.231 Dest=[::192.168.1.231]:1094
++++++ File system initialization started.
=====> all.role server
=====> ofs.trace all -debug
++++++ Storage system initialization started.
=====> all.export /
=====> oss.localroot /vol
=====> oss.trace all -debug
Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.localroot /vol
oss.trace 7ff
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 2068011008
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path / r/w nocheck nodread nomig nopurge norcreate nostage
------ Storage system initialization completed.
++++++ Configuring server role. . .
=====> all.manager 192.168.1.103:3121
=====> all.adminpath /var/spool/xrootd
=====> cms.trace all -debug
++++++ Checkpoint initialization started.
++++++ Checkpoint initialization completed.
Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration:
all.role server
ofs.maxdelay 60
ofs.persist manual hold 600 logdir /var/spool/xrootd/clustered/.ofs/posc.log
ofs.trace 7fff
------ File system server initialization completed.
Config asynchronous I/O has been disabled!
211203 19:40:45 1389 Xrd_Sched: scheduling xroot protocol anchor in 3600 seconds
211203 19:40:45 1389 Xrd_Sched: scheduling transit protocol anchor in 3600 seconds
211203 19:40:45 1401 cms_Open: Unable to connect socket to /var/spool/xrootd/clustered/.olb/olbd.admin; connection refused
211203 19:40:45 1389 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
------ xrootd clustered@xdata1-af575:1094 initialization completed.
211203 19:40:45 1394 Xrd_Sched: running main accept inq=0
211203 19:40:55 1401 cms_Finder: Connected to cmsd via /var/spool/xrootd/clustered/.olb/olbd.admin
```
</details>
Thanks in advance and kind regards,
Ewgenij
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/xrootd/xrootd/issues/1565
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
