 |
 |
I can confirm that: schroete@hc03:~$ cp MetopA.sh /mnt/xrootd/schroete/willi.dat cp: reguläre Datei '/mnt/xrootd/schroete/willi.dat' kann nicht angelegt werden: Keine Berechtigung schroete@hc03:~$ xrdfs glogin1 locate -r /xrootd/schroete/willi.dat [ERROR] Server responded with an error: [3011] No servers have the file schroete@hc03:~$ xrdfs glogin1 rm /xrootd/schroete/willi.dat [ERROR] Server responded with an error: [3011] No servers are available to write the file. schroete@hc03:~$ xrdcp MetopA.sh xroot://glogin1//xrootd/schroete/willi.dat [226B/226B][100%][===============================][226B/s] "And we _believe_ that in a C problem" Do you mean that it could be eventually a problem within C ? Hui, thats coming on strong ... So, at least we have a work around for now and i can carry with the intro of xrootd for our data system. Hopefully there can be found a real solution to this issue. If i can be of any help with testing, i still do run the test setup for a couple of days/weeks. Best Heiko Am 03.05.2018 um 09:38 schrieb Yang, Wei: > Xrdfs redirector locate -r file will also release the redirector's memory. And we _believe_ that in a C problem, you can do a straight open() (which will fail) to clear the redirector's memory. > > -- > Wei Yang | [log in to unmask] | 650-926-3338(O) > > ?-----Original Message----- > From: Heiko Schröter <[log in to unmask]> > Date: Thursday, May 3, 2018 at 12:36 AM > To: Wei Yang <[log in to unmask]>, xrootd-l <[log in to unmask]> > Cc: "[log in to unmask]" <[log in to unmask]> > Subject: Re: xrootd 4.8.3-rc1 FUSE mount "eligible servers shunned" > > Hello Wei, > > thanks for your effort in this issue. I can confirm your observations > with our simple setup. > > > STAT does not free the file name. > > schroete@CLIENT:~$ cp test.txt /mnt/xrootd/schroete/willi.dat > cp: reguläre Datei '/mnt/xrootd/schroete/willi.dat' kann nicht angelegt > werden: Keine Berechtigung > (i.e. no permissions) > > schroete@CLIENT:~$ xrdfs REDIRECTOR stat /xrootd/schroete/willi.dat > [ERROR] Server responded with an error: [3011] No servers are available > to read the file. > > schroete@CLIENT:~$ xrdfs REDIRECTOR rm /xrootd/schroete/willi.dat > [ERROR] Server responded with an error: [3011] Unable to write file; > eligible servers shunned. > > > > CAT releases the file name. > > schroete@CLIENT:~$ cp test.txt /mnt/xrootd/schroete/willi.dat > cp: reguläre Datei '/mnt/xrootd/schroete/willi.dat' kann nicht angelegt > werden: Keine Berechtigung > > schroete@CLIENT:~$ xrdfs REDIRECTOR cat /xrootd/schroete/willi.dat > [ERROR] Server responded with an error: [3011] No servers are available > to read the file. > > schroete@CLIENT:~$ xrdfs REDIRECTOR rm /xrootd/schroete/willi.dat > [ERROR] Server responded with an error: [3011] No servers are available > to write the file. > > schroete@CLIENT:~$ xrdcp test.txt > xroot://REDIRECTOR//xrootd/schroete/willi.dat > [226B/226B][100%][=======================][226B/s] > > > I did increase the cms.fxhold to 1hour. Basically we will run a WORM > system for satellite data. > > Best > Heiko > > > Am 02.05.2018 um 23:52 schrieb Yang, Wei: > > Just talked to Andy on this. He is looking at the code. On the other hand, setting cms.fxhold to something so low will overwhelm your director. Can you try > > > > xrdfs redirector stat file > > xrdfs redirector cat file > > > > We believe both will should help erase the memory in the redirector cache. However, in my (complex) setup, only the second one works. > > > > -- > > Wei Yang | [log in to unmask] | 650-926-3338 (O) > > > > On 5/2/18, 3:42 AM, "Heiko Schröter" <[log in to unmask]> wrote: > > > > Hello Wei, > > > > the param "cms.fxhold" has an influence on this issue. > > > > When i set this "cms.fxhold" to 60s (for example) i can reuse the > > offending file name after this cache timeout. > > > > I know that this imposes a possible additional load on the redirector > > but for now i could live with it until a proper solution [is|might be] > > available. > > > > Best > > Heiko > > > > > > > > > > Am 28.04.2018 um 09:48 schrieb Yang, Wei: > > > Hi Heiko, > > > > > > I understand this problem, and I don?t know if this problem has been resolved or not. You probably don?t even need xrootdfs to create this problem. I will check with Andy next week. > > > > > > Also note that the ?sss? key to be used between xrootdfs and xrootd servers is quite special (see xrootdfs man page). At the xrootdfs side, the keytab has to be created for user "anybody" and group "usrgroup", and can not have any other user/group. At the xrootd servers side, the keytab should also have this key but can have other keys as well. > > > > > > regards, > > > -- > > > Wei Yang | [log in to unmask] | 650-926-3338(O) > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Heiko Schröter <[log in to unmask]> > > > Date: Saturday, April 28, 2018 at 12:11 AM > > > To: Wei Yang <[log in to unmask]>, xrootd-l <[log in to unmask]> > > > Cc: "[log in to unmask]" <[log in to unmask]> > > > Subject: Re: xrootd 4.8.3-rc1 FUSE mount "eligible servers shunned" > > > > > >> We have created a xrootd storage system with one redirector and 12 data > > >> server. In our institut anyone needs at least read access to this system > > >> of satellite data. > > >> To protect users and the admins the FUSE mount shall be ro only because > > >> any user could use this interface on his local machine. > > >> > > >> Therefore we do use UNIX auth and it does what we had in mind. The users > > >> are recognized via ldap and can write (xrdcp et al, not FUSE cp) into > > >> their dirs which are enabled in the Authfile. > > >> > > >> > > >> Our Test Authfile: > > >> u * /xrootd lr > > >> u root /xrootd lr > > >> u schroete /xrootd/schroete a > > >> u vountas /xrootd/vountas a > > >> u schell /xrootd/schell a > > >> u bram /xrootd/bram a > > >> > > >> The Test Xrootd.cf: > > >> xrd.timeout hail 30 idle 0 kill 3 read 5 > > >> all.export /xrootd > > >> set xrdr=RDIRECTOR > > >> set inventory=/var/log/xrootd/inventory > > >> all.manager $(xrdr):3121 > > >> cms.allow host *.our.domain > > >> > > >> if $(xrdr) && named cns > > >> all.export $(inventory) > > >> xrd.port 1095 > > >> else if $(xrdr) > > >> all.role manager > > >> oss.defaults rw > > >> xrd.port 1094 > > >> else > > >> all.role server > > >> ofs.notify closew create mkdir mv rm rmdir trunc | > > >> /usr/bin/XrdCnsd -d -D 2 -i 90 -b $(xrdr):1095:$(inventory) > > >> ofs.notifymsg create $TID create $FMODE $LFN?$CGI > > >> ofs.notifymsg closew $TID closew $LFN $FSIZE > > >> xrootd.seclib /usr/lib/libXrdSec.so > > >> # sec.protocol /usr/lib sss -s /etc/xrootd/sss.keytab > > >> sec.protocol /usr/lib unix > > >> acc.authdb /etc/xrootd/Authfile > > >> acc.authrefresh 60 > > >> ofs.authorize > > >> cms.space min 100g 110g > > >> fi > > >> > > >> > > >> The FUSE mount on our number cruncher(s) is done via autofs. This is > > >> what most users would also use on their machines or a fixed mount in fstab. > > >> > > >> What happens if user "schroete" is trying to copy a file via the FUSE > > >> mount into "/xrootd/schroete": > > >> > > >> 1) schroete@client1 # cp foo.txt /mnt/xrootd/schroete/foof.txt -> > > >> Permission denied, ok. (No user shall have write access via FUSE) > > >> > > >> 2) schroete@client1 # xrdfs REDIRECTOR rm /mnt/xrootd/bram/foof.txt -> > > >> eligible server shunned > > >> > > >> The filename "foo.txt" is blocked for any further operation. xrootd > > >> claims "file exist". But this is only inside the xrootd deamons. There > > >> exisists no file on the data servers. > > >> > > >> I just rechecked it with the SSS option enabled and this Test sss.keytab > > >> but to no avail. > > >> > > >> 0 u:vountas g:users n:nowhere N:6549388083412860932 c:1524898243 e:0 f:0 > > >> k:9247585b2e1153a73b83c09c17afeb3cca432a12e3ef465634eed3e2c37f1800 > > >> 0 u:schroete g:users n:nowhere N:6549387937383972866 c:1524898209 e:0 > > >> f:0 k:9fe81373b803b3ebf86c6074a4d921703802fcd5a51030f5b95f87f7e97a9e06 > > >> 0 u:xrootd g:nogroup n:nowhere N:6549040822422077441 c:1524817390 e:0 > > >> f:0 k:12f2a7a111e4e27204224a7521a85ab3d2435d88d57a4b62a10bbd3a8b3a32c0 > > >> 0 u:bram g:users n:nowhere N:6549388053348089859 c:1524898236 e:0 f:0 > > >> k:e621083a0c29307de036dbf59a9adfa2cb40a2fde382977c2fbf99e696e63729 > > >> 0 u:schell g:users n:nowhere N:6549388113477632005 c:1524898250 e:0 f:0 > > >> k:d57d9106ad63c98eac319ab0e7515a805041bdc6a655c7c85afce37270ac74c6 > > >> > > >> > > >> Best > > >> Heiko > > >> > > >> > > >> Am 27.04.2018 um 19:51 schrieb Yang, Wei: > > >>> I am still not sure I fully understand the issue so bear with me if I am wrong: > > >>> > > >>> for xrootdfs, "sss" is optional. Without sss, the xrootdfs will access data in the xrootd cluster as the uid that runs xrootdfs, not the user that access through xrootdfs. With "sss" correctly setup (on both xrootdfs and xrootd cluster), xrootd cluster "sees" the actual user. > > >>> > > >>> If you configure your xrootd cluster to support both "sss" and "unix", then those users who don't have the "sss" key will access the xrootd cluster using "unix". So they can still read/write if you permit so (it is just not really secure). > > >>> > > >>> -- > > >>> Wei Yang | [log in to unmask] | 650-926-3338 > > > > > > -- > > ----------------------------------------------------------------------- > > Dipl.-Ing. Heiko Schröter > > Institute of Environmental Physics (IUP) phone: ++49-(0)421-218-62092 > > Institute of Remote Sensing (IFE) fax: ++49-(0)421-218-62070 > > University of Bremen (FB1) > > P.O. Box 330440 email: [log in to unmask] > > Otto-Hahn-Allee 1 > > 28359 Bremen > > Germany > > ----------------------------------------------------------------------- > > > > > > > > -- > ----------------------------------------------------------------------- > Dipl.-Ing. Heiko Schröter > Institute of Environmental Physics (IUP) phone: ++49-(0)421-218-62092 > Institute of Remote Sensing (IFE) fax: ++49-(0)421-218-62070 > University of Bremen (FB1) > P.O. Box 330440 email: [log in to unmask] > Otto-Hahn-Allee 1 > 28359 Bremen > Germany > ----------------------------------------------------------------------- > > > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > -- ----------------------------------------------------------------------- Dipl.-Ing. Heiko Schröter Institute of Environmental Physics (IUP) phone: ++49-(0)421-218-62092 Institute of Remote Sensing (IFE) fax: ++49-(0)421-218-62070 University of Bremen (FB1) P.O. Box 330440 email: [log in to unmask] Otto-Hahn-Allee 1 28359 Bremen Germany ----------------------------------------------------------------------- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1