 |
 |
Sounds like it could be it. "SSS" keying is not really an option for us because an unaware user might still mount xrootd FUSE on her own machine without it. So the root cause would remain to the system. Is there anything else i could test because we still have the test setup in work ? Heiko Am 27.04.2018 um 08:53 schrieb Yang, Wei: > In XrootdFS source code (XrdFfsXrootdfs.cc) I have the following comment which seems to describe > a scenario that is similar to what you saw. > > /* > Around May 2008, the O_EXCL was added to the _open(). No reason was given. It is removed again > due to the following reason (the situation that redirector thinks a file exist while it doesn't): > > > 1. FUSE will use _getattr to determine file status. _mknod() will be called only if _getattr() > determined that the file does not exist. > 2. In the case that rootd security is enabled, if a user create a file at an unauthorized path > (and fail), redirector thinks the files exist but it actually does't exist (enabling security > on redirector doesn't seems to help. An authorized user won't be able to create the same file > until the redirector forgets about it. > > > res = XrdFfsPosix_open(rootpath, O_CREAT | O_EXCL | O_WRONLY, S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH); > */ > > But the above comment was put there many years ago and thing may have change. And indeed I can not repeat > the problem now. In my setup which I enabled ³sss² security, is here what I see: > > > > [scs@griddev01 atlas]$ id > uid=15001(scs) gid=1051(sf) groups=1051(sf) > [scs@griddev01 atlas]$ pwd > /xrootd/atlas > [scs@griddev01 atlas]$ cp /tmp/README junk > cp: cannot create regular file `junk': Permission denied > > > > > The above show user ³scs² can not create the file due to permission. But below, user ³yangw² rm and create the file. The results are expected. > > [yangw@griddev01 atlas]$ xrdfs atlrdr1:11094 rm /xrootd/atlas/junk > [ERROR] Server responded with an error: [3011] No servers are available to write the file. > > [yangw@griddev01 atlas]$ xrdcp /tmp/README root://atlrdr1:11094//xrootd/atlas/junk > [998B/998B][100%][==================================================][998B/s] > > > > > > > > > -- > Wei Yang | [log in to unmask] | 650-926-3338(O) > > > > > > > > -----Original Message----- > From: Heiko Schröter <[log in to unmask]> > Date: Thursday, April 26, 2018 at 11:27 PM > To: Wei Yang <[log in to unmask]>, xrootd-l <[log in to unmask]> > Cc: "[log in to unmask]" <[log in to unmask]> > Subject: Re: xrootd 4.8.3-rc1 FUSE mount "eligible servers shunned" > >> No the file does not exist. I checked it on the data servers. >> >> The scenario is just as given from step 1 to step 3 in that order. >> First do a a failed "fuse" copy (permission denied, which is wanted in >> that dir). >> Just followed by the other two commands. >> Nothing in between. >> >> Heiko >> >> >> >> Am 27.04.2018 um 08:23 schrieb Yang, Wei: >>> Only the first one uses XrootdFS (xrootd fuse mount). for 3) do you know if the file actually exist before xrdcp? If so, you will need -f option to xrdcp. >>> >>> -- >>> Wei Yang | [log in to unmask] | 650-926-3338(O) >>> >>> >>> >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: <[log in to unmask]> on behalf of Heiko Schröter <[log in to unmask]> >>> Date: Thursday, April 26, 2018 at 1:45 AM >>> To: xrootd-l <[log in to unmask]> >>> Cc: "[log in to unmask]" <[log in to unmask]> >>> Subject: xrootd 4.8.3-rc1 FUSE mount "eligible servers shunned" >>> >>>> In short: >>>> 1) cp file via xrootd FUSE mount -> no permission # OK. Wanted behaviour. >>>> 2) xrdfs rm file -> Unable to write file; eligible servers shunned >>>> 3) xrdcp file -> Unable to create new file; file already exists. >>>> >>>> It does not happen when copying a file with xrdcp into a forbidden >>>> directory. >>>> >>>> Only solution seems to be restarting the redirector. >>>> This situation may lead into a completely blocked namespace if a user >>>> starts a mass copy job via FUSE mount (by accident). >>>> >>>> Are there any other options ? >>>> IMHO xrootd should not create any caching issues when a file is >>>> forbidden to write/create/copy. >>>> >>>> >>>> Heiko >>>> >>>> P.S: Also this "stalling client" pops up. >>>> >>>> >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++ >>>> Resume State after fresh restart of redirector: >>>> 180426 10:03:53 4079 setStatus REDIRECTOR sent resume event >>>> 180426 10:03:53 4079 cms_setStatus: Manager REDIRECTOR resumed >>>> >>>> >>>> Copy via FUSE mnt without permissions which is the wanted behaviour: >>>> schroete@CLIENT:~$ cp MetopA.sh /mnt/xrootd/schroete/willi.dat >>>> cp: reguläre Datei '/mnt/xrootd/schroete/willi.dat' kann nicht angelegt >>>> werden: Keine Berechtigung >>>> i.e. no permissions >>>> >>>> ```bash >>>> 180426 10:15:21 4071 XrdSched: Now have 3 workers >>>> 180426 10:15:21 4071 XrdSched: running main accept inq=0 >>>> 180426 10:15:21 4247 XrdXeq: Worker thread started >>>> 180426 10:15:21 4072 XrdInet: Accepted connection from 7@CLIENT >>>> 180426 10:15:21 4072 XrdProtocol: matched protocol xrootd >>>> 180426 10:15:21 4072 ?:7@CLIENT XrdPoll: FD 7 attached to poller 0; num=1 >>>> 180426 10:15:21 4072 ?:7@CLIENT XrootdProtocol: 0000 req=login dlen=104 >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdResponse: 0000 sending >>>> 16 data bytes >>>> 180426 10:15:21 4072 XrootdXeq: root.271390:7@CLIENT pvt IPv4 login >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdProtocol: 0100 >>>> req=locate dlen=8 >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdProtocol: 0100 locate >>>> /xrootd/ >>>> 180426 10:15:21 4072 root.271390:7@CLIENT ofs_fsctl: fn=/xrootd/ >>>> 180426 10:15:21 4079 Receive glogin1 4 bytes on 3071 >>>> 180426 10:15:21 4079 Decode glogin1 delays root.271390:7@CLIENT 5 /xrootd/ >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdProtocol: 0100 rc=5 >>>> locate /xrootd/ >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdProtocol: 0100 stalling >>>> client for 5 sec >>>> 180426 10:15:21 4072 root.271390:7@CLIENT XrootdResponse: 0100 sending 4 >>>> data bytes; status=4005 >>>> 180426 10:15:23 4247 XrdSched: Now have 4 workers >>>> 180426 10:15:23 4247 XrdSched: running underused thread monitor inq=0 >>>> 180426 10:15:23 4247 XrdSched: 4 threads; 0 idle >>>> 180426 10:15:23 4247 XrdSched: scheduling underused thread monitor in >>>> 780 seconds >>>> 180426 10:15:23 4249 XrdXeq: Worker thread started >>>> 180426 10:15:24 4072 root.271390:7@CLIENT XrootdProtocol: 0100 request >>>> timeout; read 0 of 24 bytes >>>> 180426 10:15:24 4072 XrdPoll: Poller 0 enabled root.271390:7@CLIENT >>>> 180426 10:15:26 4247 XrdSched: running root.271390:7@CLIENT inq=0 >>>> 180426 10:15:26 4247 root.271390:7@CLIENT XrootdProtocol: 0100 >>>> req=locate dlen=8 >>>> 180426 10:15:26 4247 root.271390:7@CLIENT XrootdProtocol: 0100 locate >>>> /xrootd/ >>>> 180426 10:15:26 4247 root.271390:7@CLIENT ofs_fsctl: fn=/xrootd/ >>>> 180426 10:15:26 4079 Receive glogin1 315 bytes on 4095 >>>> 180426 10:15:26 4079 Decode glogin1 sent root.271390:7@CLIENT >>>> 'Sw[::192.168.16.146]:1094 Sw[::192.168.16.144]:1094 >>>> Sw[::192.168.16.127]:1094 Sw[::192.168.16.120]:1094 >>>> Sw[::192.168.16.147]:1094 Sw[::192.168.16.139]:1094 >>>> Sw[::192.168.16.97]:1094 Sw[::192.168.16.195]:1094 >>>> Sw[::192.168.16.196]:1094 Sw[::192.168.16.134]:1094 >>>> Sw[::192.168.16.121]:1094 Sw[::192.168.16.217]:1094' /xrootd/ >>>> 180426 10:15:26 4247 root.271390:7@CLIENT XrootdProtocol: 0100 rc=-1024 >>>> locate /xrootd/ >>>> 180426 10:15:26 4247 root.271390:7@CLIENT XrootdResponse: 0100 sending >>>> 311 data bytes >>>> 180426 10:15:26 4249 XrdSched: running main accept inq=0 >>>> 180426 10:15:26 4071 XrdInet: Accepted connection from 19@CLIENT >>>> 180426 10:15:26 4071 XrdProtocol: matched protocol xrootd >>>> 180426 10:15:26 4071 ?:19@CLIENT XrdPoll: FD 19 attached to poller 1; num=1 >>>> 180426 10:15:26 4071 ?:19@CLIENT XrootdProtocol: 0000 req=login dlen=104 >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT XrootdResponse: 0000 sending >>>> 16 data bytes >>>> 180426 10:15:26 4071 XrootdXeq: h21.271390:19@CLIENT pvt IPv4 login >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT XrootdProtocol: 0100 req=open >>>> dlen=27 >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT XrootdProtocol: 0100 open >>>> udmat /xrootd//schroete/willi.dat >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT ofs_open: 200-40644 >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:15:26 4079 Receive glogin1 4 bytes on 5119 >>>> 180426 10:15:26 4079 Decode glogin1 delays h21.271390:19@CLIENT 5 >>>> /xrootd/schroete/willi.dat >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT XrootdProtocol: 0100 stalling >>>> client for 5 sec >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT XrootdResponse: 0100 sending 4 >>>> data bytes; status=4005 >>>> 180426 10:15:26 4071 h21.271390:19@CLIENT ofs_close: use=0 fn=dummy >>>> 180426 10:15:29 4247 root.271390:7@CLIENT XrootdProtocol: 0100 request >>>> timeout; read 0 of 24 bytes >>>> 180426 10:15:29 4247 XrdPoll: Poller 0 enabled root.271390:7@CLIENT >>>> 180426 10:15:29 4071 h21.271390:19@CLIENT XrootdProtocol: 0100 request >>>> timeout; read 0 of 24 bytes >>>> 180426 10:15:29 4071 XrdPoll: Poller 1 enabled h21.271390:19@CLIENT >>>> 180426 10:15:31 4072 XrdSched: running h21.271390:19@CLIENT inq=0 >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT XrootdProtocol: 0100 req=open >>>> dlen=27 >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT XrootdProtocol: 0100 open >>>> udmat /xrootd//schroete/willi.dat >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT ofs_open: 200-40644 >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:15:31 4079 Receive glogin1 19 bytes on 6143 >>>> 180426 10:15:31 4079 Decode glogin1 redirects h21.271390:19@CLIENT to >>>> 192.168.16.217:1094 /xrootd/schroete/willi.dat >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT XrootdProtocol: 0100 >>>> redirecting to 192.168.16.217:1094 >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT XrootdResponse: 0100 sending >>>> 18 data bytes; status=4004 >>>> 180426 10:15:31 4072 h21.271390:19@CLIENT ofs_close: use=0 fn=dummy >>>> 180426 10:15:34 4072 h21.271390:19@CLIENT XrootdProtocol: 0100 request >>>> timeout; read 0 of 24 bytes >>>> 180426 10:15:34 4072 XrdPoll: Poller 1 enabled h21.271390:19@CLIENT >>>> ``` >>>> >>>> Trying to remove the file: >>>> schroete@CLIENT:~$ xrdfs glogin1 rm /xrootd/schroete/willi.dat >>>> [ERROR] Server responded with an error: [3011] Unable to write file; >>>> eligible servers shunned. >>>> >>>> ```bash >>>> 180426 10:20:15 4247 XrdSched: running main accept inq=0 >>>> 180426 10:20:15 4249 XrdInet: Accepted connection from 18@CLIENT >>>> 180426 10:20:15 4249 XrdProtocol: matched protocol xrootd >>>> 180426 10:20:15 4249 ?:18@CLIENT XrdPoll: FD 18 attached to poller 2; num=1 >>>> 180426 10:20:15 4249 ?:18@CLIENT XrootdProtocol: 0000 req=login dlen=101 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdResponse: 0000 >>>> sending 16 data bytes >>>> 180426 10:20:15 4249 XrootdXeq: schroete.271507:18@CLIENT pvt IPv4 login >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdProtocol: 0100 >>>> req=rm dlen=26 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT ofs_remove: f >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:20:15 4079 Receive glogin1 19 bytes on 7167 >>>> 180426 10:20:15 4079 Decode glogin1 redirects schroete.271507:18@CLIENT >>>> to 192.168.16.217:1094 /xrootd/schroete/willi.dat >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdProtocol: 0100 >>>> rc=-256 rm /xrootd/schroete/willi.dat >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdProtocol: 0100 >>>> redirecting to 192.168.16.217:1094 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdResponse: 0100 >>>> sending 18 data bytes; status=4004 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdProtocol: 0100 >>>> req=rm dlen=62 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT ofs_remove: f >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:20:15 4079 Receive glogin1 52 bytes on 8191 >>>> 180426 10:20:15 4079 Decode glogin1 gave schroete.271507:18@CLIENT err >>>> -2 'Unable to write file; eligible servers shunned.' >>>> /xrootd/schroete/willi.dat >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdProtocol: 0100 >>>> rc=-1 rm /xrootd/schroete/willi.dat >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrootdResponse: 0100 >>>> sending err 3011: Unable to write file; eligible servers shunned. >>>> 180426 10:20:15 4249 XrootdXeq: schroete.271507:18@CLIENT disc 0:00:00 >>>> 180426 10:20:15 4249 schroete.271507:18@CLIENT XrdPoll: FD 18 detached >>> >from poller 2; num=0 >>>> ``` >>>> >>>> >>>> Tryint to copy with xrdcp tool: >>>> schroete@CLIENT:~$ xrdcp MetopA.sh >>>> xroot://glogin1//xrootd/schroete/willi.dat >>>> [0B/0B][100%][==================================================][0B/s] >>>> Run: [ERROR] Server responded with an error: [3011] Unable to create new >>>> file; file already exists.180426 >>>> >>>> ```bash >>>> 180426 10:21:03 4071 XrdSched: running h21.271390:19@CLIENT inq=0 >>>> 180426 10:21:03 4071 XrootdXeq: h21.271390:19@CLIENT disc 0:05:37 >>>> 180426 10:21:03 4071 h21.271390:19@CLIENT XrdPoll: FD 19 detached from >>>> poller 1; num=0 >>>> 180426 10:21:03 4072 XrdSched: running root.271390:7@CLIENT inq=0 >>>> 180426 10:21:03 4072 XrootdXeq: root.271390:7@CLIENT disc 0:05:42 >>>> 180426 10:21:03 4072 root.271390:7@CLIENT XrdPoll: FD 7 detached from >>>> poller 0; num=0 >>>> 180426 10:21:05 4247 XrdInet: Accepted connection from 21@CLIENT >>>> 180426 10:21:05 4247 XrdProtocol: matched protocol xrootd >>>> 180426 10:21:05 4247 ?:21@CLIENT XrdPoll: FD 21 attached to poller 0; num=1 >>>> 180426 10:21:05 4247 ?:21@CLIENT XrootdProtocol: 0000 req=login dlen=101 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdResponse: 0000 >>>> sending 16 data bytes >>>> 180426 10:21:05 4247 XrootdXeq: schroete.271528:21@CLIENT pvt IPv4 login >>>> 180426 10:21:05 4249 XrdSched: running main accept inq=0 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> req=stat dlen=26 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT ofs_stat: >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:21:05 4079 Receive glogin1 19 bytes on 9215 >>>> 180426 10:21:05 4079 Decode glogin1 redirects schroete.271528:21@CLIENT >>>> to 192.168.16.217:1094 /xrootd/schroete/willi.dat >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> rc=-256 stat /xrootd/schroete/willi.dat >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> redirecting to 192.168.16.217:1094 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdResponse: 0100 >>>> sending 18 data bytes; status=4004 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> req=stat dlen=62 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT ofs_stat: >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:21:05 4079 Receive glogin1 47 bytes on 10239 >>>> 180426 10:21:05 4079 Decode glogin1 gave schroete.271528:21@CLIENT err >>>> -2 'No servers are available to read the file.' /xrootd/schroete/willi.dat >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> rc=-1 stat /xrootd/schroete/willi.dat >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdResponse: 0100 >>>> sending err 3011: No servers are available to read the file. >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> req=open dlen=40 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 open >>>> unmat /xrootd/schroete/willi.dat?oss.asize=226 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT ofs_open: 102-40644 >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:21:05 4079 Receive glogin1 4 bytes on 11263 >>>> 180426 10:21:05 4079 Decode glogin1 delays schroete.271528:21@CLIENT 5 >>>> /xrootd/schroete/willi.dat >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> stalling client for 5 sec >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT XrootdResponse: 0100 >>>> sending 4 data bytes; status=4005 >>>> 180426 10:21:05 4247 schroete.271528:21@CLIENT ofs_close: use=0 fn=dummy >>>> 180426 10:21:08 4247 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> request timeout; read 0 of 24 bytes >>>> 180426 10:21:08 4247 XrdPoll: Poller 0 enabled schroete.271528:21@CLIENT >>>> 180426 10:21:10 4071 XrdSched: running schroete.271528:21@CLIENT inq=0 >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT XrootdProtocol: 0100 >>>> req=open dlen=40 >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT XrootdProtocol: 0100 open >>>> unmat /xrootd/schroete/willi.dat?oss.asize=226 >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT ofs_open: 102-40644 >>>> fn=/xrootd/schroete/willi.dat >>>> 180426 10:21:10 4079 Receive glogin1 52 bytes on 12287 >>>> 180426 10:21:10 4079 Decode glogin1 gave schroete.271528:21@CLIENT err >>>> -2 'Unable to create new file; file already exists.' >>>> /xrootd/schroete/willi.dat >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT XrootdResponse: 0100 >>>> sending err 3011: Unable to create new file; file already exists. >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT ofs_close: use=0 fn=dummy >>>> 180426 10:21:10 4071 XrootdXeq: schroete.271528:21@CLIENT disc 0:00:05 >>>> 180426 10:21:10 4071 schroete.271528:21@CLIENT XrdPoll: FD 21 detached >>> >from poller 0; num=0 >>>> 180426 10:22:23 4069 XrdBuffManager: Reshaper has 3K; target 403877K >>>> ``` >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> >> -- >> ----------------------------------------------------------------------- >> Dipl.-Ing. Heiko Schröter >> Institute of Environmental Physics (IUP) phone: ++49-(0)421-218-62092 >> Institute of Remote Sensing (IFE) fax: ++49-(0)421-218-62070 >> University of Bremen (FB1) >> P.O. Box 330440 email: [log in to unmask] >> Otto-Hahn-Allee 1 >> 28359 Bremen >> Germany >> ----------------------------------------------------------------------- >> -- ----------------------------------------------------------------------- Dipl.-Ing. Heiko Schröter Institute of Environmental Physics (IUP) phone: ++49-(0)421-218-62092 Institute of Remote Sensing (IFE) fax: ++49-(0)421-218-62070 University of Bremen (FB1) P.O. Box 330440 email: [log in to unmask] Otto-Hahn-Allee 1 28359 Bremen Germany ----------------------------------------------------------------------- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1