 |
 |
So it may be the proxy. CCing the xrootd developers list. Lukasz On 31.05.2013 15:44, Mol, Xavier (SCC) wrote: > Hi Lukasz, > > from the perspective of dCache, today either a transfer for /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root (which equals /pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/user/ivukotic/a6/57/group.test.hc.NTUP_SMWZ.root which equals 0000000BDA41933C4E8A8EC629636A4F3624 for dCache) succeeded regularly, or this error occurred: > > "No connection from Xrootd client after 300 seconds. Giving up." > > Ciao, > Xavier. > >> -----Original Message----- >> From: Lukasz Janyst [mailto:[log in to unmask]] >> Sent: Friday, May 31, 2013 3:29 PM >> To: Mol, Xavier (SCC) >> Cc: Ilija Vukotic; Petzold, Andreas (SCC); Lukasz Janyst; [log in to unmask]; Lincoln Bryant; Wei Yang; dcache- >> [log in to unmask] >> Subject: Re: [dcache-admin] Re: default ReadCacheSize value and other questions >> >> Hi Xavier, >> >> I don't thinks so. I have no experience with dCache and this kind of >> setups, but looking at your configs it's either a problem in the proxy >> code (Wei, could you test it somehow?) or in dCache itself. >> >> Cheers, >> Lukasz >> >> On 31.05.2013 15:07, Mol, Xavier (SCC) wrote: >>> Hi Lukasz, >>> >>> this setup is following the instructions for beeing a proxy: https://twiki.cern.ch/twiki/bin/view/Atlas/FAXDECloud. As far as I know, >> we only deviated from that guide with regards to the GSI security configuration - we tried to grant dteam users (read) access >> additionally to ATLAS clients (https://twiki.cern.ch/twiki/bin/view/Atlas/XrootdVOMSsecPlugin). This should be unrelated to your >> issues, right? >>> >>> Ciao, >>> Xavier. >>> >>>> -----Original Message----- >>>> From: Lukasz Janyst [mailto:[log in to unmask]] >>>> Sent: Friday, May 31, 2013 2:40 PM >>>> To: Ilija Vukotic >>>> Cc: Petzold, Andreas (SCC); Lukasz Janyst; [log in to unmask]; Lincoln Bryant; Wei Yang; dcache- >>>> [log in to unmask] >>>> Subject: [dcache-admin] Re: default ReadCacheSize value and other questions >>>> >>>> Hi Andreas, >>>> >>>> The server behaves incorrectly and erratically when presented with a >>>> request for a chunk that exceeds the file boundary. Also, it sometimes >>>> refuses to open a file that it has stated correctly. Please see the >>>> attached test case. What kind of setup are you running on that box? Is >>>> it a proxy, standard xrootd, something else? >>>> >>>> With the attached test program, I get the following on lxplus: >>>> >>>> ]==> ./a.out >>>> File size: 797152257 >>>> Read 16777216 bytes at offset 788529152 >>>> Unable to read: [ERROR] Server responded with an error: [3011] Unable to >>>> read >>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>> no such file or directory >>>> >>>> ]==> ./a.out >>>> File size: 797152257 >>>> Read 16777216 bytes at offset 788529152 >>>> Status of all operations: OK >>>> Received bytes: 8388608 >>>> Should have received: 8623105 >>>> >>>> ]==> ./a.out >>>> File size: 797152257 >>>> Unable to open: [ERROR] Server responded with an error: [3005] Unable to >>>> open >>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>> operation canceled >>>> >>>> In the case where read returned the OK response but incorrect chunk, >>>> xrdcopy should have been able to detect the error. I will make sure it does. >>>> >>>> Cheers, >>>> Lukasz >>>> >>>> >>>> On 30.05.2013 20:35, Ilija Vukotic wrote: >>>>> Hi Andreas, >>>>> >>>>> would you mind trying the same command from lxplus? >>>>> >>>>> Ilija >>>>> >>>>> -- >>>>> Ilija Vukotic • Skype ivukotic • +1 872 230 6435 • University of Chicago >>>>> >>>>> >>>>> >>>>> On May 30, 2013, at 13:32 , Andreas Petzold <[log in to unmask]> >>>>> wrote: >>>>> >>>>>> Hi Ilija, >>>>>> >>>>>> yes, in both cases I was using 3.3.2. >>>>>> >>>>>> Andreas >>>>>> >>>>>> >>>>>> On 05/30/2013 08:25 PM, Ilija Vukotic wrote: >>>>>>> Hi, >>>>>>> Was the client 3.3.2? >>>>>>> >>>>>>> ilija >>>>>>> >>>>>>> Thumb typed >>>>>>> >>>>>>> ----- Reply message ----- >>>>>>> From: "Andreas Petzold" <[log in to unmask]> >>>>>>> To: "Ilija Vukotic" <[log in to unmask]> >>>>>>> Cc: "Lukasz Janyst" <[log in to unmask]>, >>>>>>> "[log in to unmask]" >>>>>>> <[log in to unmask]>, "Lincoln Bryant" >>>>>>> <[log in to unmask]>, "Wei Yang" <[log in to unmask]> >>>>>>> Subject: default ReadCacheSize value and other questions >>>>>>> Date: Thu, May 30, 2013 11:54 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi Ilja, >>>>>>> >>>>>>> FYI, I'm able to run xrdcopy successfully for your file on my desktop at >>>>>>> KIT and on lxplus. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Andreas >>>>>>> >>>>>>> On 05/30/2013 06:04 PM, Ilija Vukotic wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I add Andreas and Guenter. Lincoln attempted to xrdcopy file form FZK today between 17:00 and 18:00 (your time) and failed. >>>> We would like to know what is in the servers log. Would you mind looking it up? >>>>>>>> (ordinary xrdcp works ok, but xrdcopy always fails from your site.) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ilija >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On May 30, 2013, at 10:57 , Lukasz Janyst <[log in to unmask]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, this is a message that the client gets from the server. Can you actually check what is happening server-side? >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Lukasz >>>>>>>>> >>>>>>>>> On 30.05.2013 17:55, Lincoln Bryant wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I also get this error. >>>>>>>>>> >>>>>>>>>> [lincolnb@uct2-int ~]$ xrdcopy --streams 1 root://f01-140-115- >>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root - > /dev/null >>>>>>>>>> [ 98%][=================================================>] [752M/760M] >>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] Unable to read >>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; no such file or directory >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [lincolnb@uct2-int ~]$ rpm -q xrootd-client >>>>>>>>>> xrootd-client-3.3.2-1.el5 >>>>>>>>>> >>>>>>>>>> --Lincoln >>>>>>>>>> >>>>>>>>>> On May 23, 2013, at 9:36 AM, Ilija Vukotic wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I added Wei and Lincoln to this mail so they can try too. >>>>>>>>>>> >>>>>>>>>>> I've just retried with both xrdcp and xrdcopy in a virgin shell on lxplus and still get the same behavior: >>>>>>>>>>> >>>>>>>>>>> ~ >xrdcopy --streams 1 root://f01-140-115- >>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root - > /dev/null >>>>>>>>>>> [ 98%][=================================================>] [752M/760M] >>>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] Unable to read >>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; no such file or directory >>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>> xrdcp --streams 1 root://f01-140-115- >>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root - > /dev/null >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Ilija >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Nullius in verba >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On May 23, 2013, at 2:07 , Lukasz Janyst <[log in to unmask]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I get this error with both clients. >>>>>>>>>>>> >>>>>>>>>>>> Lukasz >>>>>>>>>>>> >>>>>>>>>>>> On 23.05.2013 09:05, Lukasz Janyst wrote: >>>>>>>>>>>>> Hi Ilija, >>>>>>>>>>>>> >>>>>>>>>>>>> I constantly get: >>>>>>>>>>>>> >>>>>>>>>>>>> ]==> xrdcopy --streams 1 >>>>>>>>>>>>> root://f01-140-115- >> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>>>>>>>>>>>> - > /dev/null >>>>>>>>>>>>> [100%][==================================================] [0/0] >>>>>>>>>>>>> Run: [ERROR] Server responded with an error: [3005] Unable to open >>>>>>>>>>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>>>>>>>>>> operation canceled >>>>>>>>>>>>> >>>>>>>>>>>>> Are you sure the server config is OK? >>>>>>>>>>>>> >>>>>>>>>>>>> Lukasz >>>>>>>>>>>>> >>>>>>>>>>>>> On 23.05.2013 02:20, Ilija Vukotic wrote: >>>>>>>>>>>>>> Hi Lukasz, >>>>>>>>>>>>>> >>>>>>>>>>>>>> when doing this: >>>>>>>>>>>>>> xrdcopy --streams 1 >>>>>>>>>>>>>> root://f01-140-115- >> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>>>>>>>>>>>>> - > /dev/null >>>>>>>>>>>>>> from lxplus it always (!) fails between 94% and 98% in one of the two >>>>>>>>>>>>>> ways. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ 96%][================================================> ] [736M/760M] >>>>>>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] Unable to read >>>>>>>>>>>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>>>>>>>>>>> no such file or directory >>>>>>>>>>>>>> [ 98%][=================================================>] [748M/760M] >>>>>>>>>>>>>> Run: [ERROR] Internal error >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> at the same time standard xrdcp works as expected. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Security is off at that server so please feel free to try and replicate. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Nullius in verba >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On May 22, 2013, at 10:47 , Ilija Vukotic <[log in to unmask]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was using standard xrdcp you get at lxplus (which is 3.3.2) >>>>>>>>>>>>>>> Now I'll repeat it all with xrdcopy. Will let you know what I measure. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Nullius in verba >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On May 22, 2013, at 8:32 , Lukasz Janyst <[log in to unmask]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Andreas has now done some testing as well: >>>>>>>>>>>>>>>> https://github.com/xrootd/xrootd/issues/20 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Lukasz >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 22.05.2013 14:34, Lukasz Janyst wrote: >>>>>>>>>>>>>>>>> Hi Ilija, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> which version of xrootd are you using? Would it be possible for you >>>>>>>>>>>>>>>>> to try the new client (xrdcopy) instead of the old one (xrdcp)? At >>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>> from version 3.3.2? We were able to get around 750MB/s with just one >>>>>>>>>>>>>>>>> client on 10GB link. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The default for the old client is indeed one stream, but if you >>>>>>>>>>>>>>>>> don't disable the cache completely it will send many requests >>>>>>>>>>>>>>>>> synchronously. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>> Lukasz >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 21.05.2013 09:00, Ilija Vukotic wrote: >>>>>>>>>>>>>>>>>> Hi Lukasz, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I was trying to find out why xrdcp is not using the full available >>>>>>>>>>>>>>>>>> bandwidth at one 10Gb WAN link and so was playing with parameters >>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>> number of streams, size of ReadCache etc. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What I don't understand is that whatever value I set for >>>>>>>>>>>>>>>>>> ReadCacheSize I >>>>>>>>>>>>>>>>>> get performance roughly 10 times worse than if I set nothing. >>>>>>>>>>>>>>>>>> Strange thing here is that performance is 10x worse even when I >>>>>>>>>>>>>>>>>> set it >>>>>>>>>>>>>>>>>> to it's default value: 4000000. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also one more thing is puzzling to me: I have a way to intercept >>>>>>>>>>>>>>>>>> messages exchanged between xrootd server and client. Looking in >>>>>>>>>>>>>>>>>> them I >>>>>>>>>>>>>>>>>> see that default behavior of xrdcp is to at the start request 20 >>>>>>>>>>>>>>>>>> chunks >>>>>>>>>>>>>>>>>> of 4MB each. Documentation claims that the default is one stream? >>>>>>>>>>>>>>>>>> Could you explain this to me or point me to some documentation I >>>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>> read? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks a lot, >>>>>>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Dr Ilija Vukotic [log in to unmask] >>>>>>>>>>>>>>>>>> University of Chicagohttp://www.vukotic.me >>>>>>>>>>>>>>>>>> 5620 S Ellis Ave Tel: +1-773-702-7475 >>>>>>>>>>>>>>>>>> Chicago IL 60637, USA >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Karlsruhe Institute of Technology (KIT) >>>>>>> Steinbuch Centre for Computing (SCC) >>>>>>> >>>>>>> Andreas Petzold >>>>>>> >>>>>>> Hermann-von-Helmholtz-Platz 1, Building 449, Room 202 >>>>>>> D-76344 Eggenstein-Leopoldshafen >>>>>>> >>>>>>> Tel: +49 721 608 24916 >>>>>>> Fax: +49 721 608 24972 >>>>>>> Email: [log in to unmask] >>>>>>> www.scc.kit.edu <http://www.scc.kit.edu> >>>>>>> >>>>>>> KIT – University of the State of Baden-Wuerttemberg and >>>>>>> National Research Center of the Helmholtz Association >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Karlsruhe Institute of Technology (KIT) >>>>>> Steinbuch Centre for Computing (SCC) >>>>>> >>>>>> Andreas Petzold >>>>>> >>>>>> Hermann-von-Helmholtz-Platz 1, Building 449, Room 202 >>>>>> D-76344 Eggenstein-Leopoldshafen >>>>>> >>>>>> Tel: +49 721 608 24916 >>>>>> Fax: +49 721 608 24972 >>>>>> Email: [log in to unmask] >>>>>> www.scc.kit.edu >>>>>> >>>>>> KIT – University of the State of Baden-Wuerttemberg and >>>>>> National Research Center of the Helmholtz Association >>>>>> >>>>> >>> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1