Hi all,

Since a few days, our redirectors are rejecting some clients with “Too many attempts to gain dfs [read|write] access to the file”. Are there any obvious causes to check when getting this error? Could this be shadowing an underlying cause that we should investigate?

The source code [0] indicates this is a client setting, but it appears that neither redirector/server (v4.10.0) nor client (v4.10.0) were changed when the error started occurring. We see the error for repeated SAM tests [1, 2] of which many are still passing, so the same setup on both sides *sometimes* fails and *sometimes* succeeds. Could there be some race condition we should take a look at?

Cheers,
Max

[0]
https://github.com/xrootd/xrootd/blob/cfdc6aca0bde74f7b65000987f846512c42a87ab/src/XrdCms/XrdCmsCluster.cc#L948

[1] Redirector Log
/var/log/xrootd/manager/cmsd.log:191015 09:08:30 30012 SelDFS seeking /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/cmsd.log:191015 09:08:36 14961 SelNode f01-120-181-e.gridka.de assigned /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/cmsd.log:191015 09:08:44 14961 SelNode f01-124-182-e.gridka.de assigned /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:31 24030 Decode f01-031-107-e delays monalisa.1782:[log in to unmask] 5 /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:36 24030 Decode f01-031-107-e gave monalisa.1782:[log in to unmask] err -2 'Unable to access file; file does not exist.' /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:36 8693 monalisa.1782:[log in to unmask] ofs_open: 100200-40644 fn=/13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:36 24030 Decode f01-031-107-e redirects monalisa.1782:[log in to unmask] to f01-120-181-e.gridka.de:1094 /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:44 8876 monalisa.1782:[log in to unmask] ofs_open: 100200-40644 fn=/13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:44 24030 Decode f01-031-107-e redirects monalisa.1782:[log in to unmask] to f01-124-182-e.gridka.de:1094 /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:52 8693 monalisa.1782:[log in to unmask] ofs_open: 100200-40644 fn=/13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a
/var/log/xrootd/manager/xrootd.log:191015 09:08:52 24030 Decode f01-031-107-e gave monalisa.1782:[log in to unmask] err -2 'Too many attempts to gain dfs write access to the file' /13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a

[2] Client operation/log
Operation ADD on storage ALICE::FZK::SE has failed with the following message:
Test finished at: Tue Oct 15 09:08:52 CEST 2019
Duration: 22s
/home/monalisa/xrootd/bin/xrdcp exited with exit code 54: [ERROR] Server responded with an error: [3011] Too many attempts to gain dfs write access to the file
Full command was:
export XRD_CONNECTIONWINDOW="3"
export XRD_CONNECTIONRETRY="1"
export XRD_TIMEOUTRESOLUTION="1"
export XRD_PREFERIPV4="1"
export XRD_REQUESTTIMEOUT="60"
/home/monalisa/xrootd/bin/xrdcp --nopbar --path --verbose --force --posc /home/monalisa/MLrepository/bin/setesting/testfile -ODauthz=-----BEGIN SEALED CIPHER-----
NDgFlf0forRXKNGHTR8t8kNTGaHbVlQdY+DaEd+pxoNiN+GFQrsa4WQILo+Fldj6oG7g+xOiYpyS
sOn5nUnGWgVjSWyQKtkEiKbfAT+vEL9t7tv6mzoSmm6ZYKnzRToKKfY0lxvI2r4E+Dpnd0r4pnYt
QPRDzwX8fiONPT7HIlc=
-----END SEALED CIPHER-----
-----BEGIN SEALED ENVELOPE-----
AAAAgGiR2Vw+5k3SF6+OinhajOJAYvZOKhzeAMkU61QUIY+qwfbMBKkLwIYkeD84bxcnyk4ylkgw
c8PSGv7k63I-qES8FZh1Ynm4+ekID0DGEYHBytVO2z8YF6t1TcPGXM5bpXu7syrLo7H+9T4Xhz7N
3o3ueFXThl7whN40ZblrRgJbe7FJ6YA6wNV6Dz62GXeUo+bcbkoilRMUpVfdC-yvrmirOv2Kb-fw
puA5+gVnGtgz0BVi3FfLCh02duQjK+GlehdwTgy53zt0daDkvu1KaEf9xbQKUkUM-vedUK-MZqL2
EieyCoIqtHD12WByDEO1BQTC72+aSUXKk+Pgrx0sLQ1kQQGbMvIBk9CpYk1yZnzYUiSGfjtCCegr
GUylNHcEi4xmIHgrLFFpBH6UvlIprQYh+qdjUZrCfhZdpPeSCCr4Oyu0f7Z17eiISz2alhT+M277
yO-W2+5NTNnhoVC8Uq7UyqehzDlpt-gQuGsBYiCy67WPxOVoDKKGymdQXyZJ6nt4FY+vHLyGAkFp
JQkLr5OD08VGmjG5Swlboka0ggHD-zlr0f5-9Owbv9sjgS56tRzxGw5zfWgZKNMvC5+2PQkk6ilx
svhDRb5UdBZa4O53DnGbyl0mSzVGL-WK2tY08RsoQrer3APY-aAwHIfkvwC0TlG-IqaTlpK8Clnb
911gECBdFv7H-8UXpLOcZhB-iOnf0tT1aLPPmrcx-SFjdF3FLTOBbzHLBSC+Jx0XHhzx4gTPudS1
DVTAT9aFG6BE3puv5bWudrKAtT-FQ2LkoYanvSvz3GsBO1Yu372Lm4lN1i12mjLDgsV2rMVtUIeV
SjpR2FqcIN7bniGRqdNGnxLDjIav9x9WIhL2M+OalngesKwYaeSZDb6oFCnghdEswYY6kyHMsoyA
OqQZxcqptTjI16Ppchgi8cDdqsdRcbUMW+4TMiPxJFFodzRQfDRc1VwsB3dhmI9AFd--q+mMNqih
gSKqFX9IH47O0vof4AyxAfLuoQJKNcRIo0cRr2BB8QzD6RXD1AKIgSdvR4uBGgk=
-----END SEALED ENVELOPE-----
root://alice-disk-se.gridka.de:1094//13/40572/97b83210-ef1a-11e9-bf12-0242a3ceac7a

[3] Redirector cms configuration
cms.delay drop 45 qdn 3 servers 2 service 30 startup 30
cms.sched affinity none cpu 30 runq 50 io 10 mem 10 space 0 fuzz 100
cms.ping 20 log 1 usage 15
cms.space min 16g 16g
cms.dfs limit 0 lookup distrib mdhold 0 redirect verify retries 2

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1