The xrdcopy will now see the mismatched size and complain: https://github.com/xrootd/xrootd/commit/06096208603569c54ce7b43a568fe05e1359a89f ]==> xrdcopy --streams 1 root://f01-140-115-e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root - > /dev/null [ 99%][=================================================>] [760M/760M] Run: [ERROR] Received corrupted data Cheers, Lukasz On 31.05.2013 16:04, Lukasz Janyst wrote: > So it may be the proxy. CCing the xrootd developers list. > > Lukasz > > On 31.05.2013 15:44, Mol, Xavier (SCC) wrote: >> Hi Lukasz, >> >> from the perspective of dCache, today either a transfer for >> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >> (which equals >> /pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/user/ivukotic/a6/57/group.test.hc.NTUP_SMWZ.root >> which equals 0000000BDA41933C4E8A8EC629636A4F3624 for dCache) >> succeeded regularly, or this error occurred: >> >> "No connection from Xrootd client after 300 seconds. Giving up." >> >> Ciao, >> Xavier. >> >>> -----Original Message----- >>> From: Lukasz Janyst [mailto:[log in to unmask]] >>> Sent: Friday, May 31, 2013 3:29 PM >>> To: Mol, Xavier (SCC) >>> Cc: Ilija Vukotic; Petzold, Andreas (SCC); Lukasz Janyst; >>> [log in to unmask]; Lincoln Bryant; Wei Yang; >>> dcache- >>> [log in to unmask] >>> Subject: Re: [dcache-admin] Re: default ReadCacheSize value and other >>> questions >>> >>> Hi Xavier, >>> >>> I don't thinks so. I have no experience with dCache and this >>> kind of >>> setups, but looking at your configs it's either a problem in the proxy >>> code (Wei, could you test it somehow?) or in dCache itself. >>> >>> Cheers, >>> Lukasz >>> >>> On 31.05.2013 15:07, Mol, Xavier (SCC) wrote: >>>> Hi Lukasz, >>>> >>>> this setup is following the instructions for beeing a proxy: >>>> https://twiki.cern.ch/twiki/bin/view/Atlas/FAXDECloud. As far as I >>>> know, >>> we only deviated from that guide with regards to the GSI security >>> configuration - we tried to grant dteam users (read) access >>> additionally to ATLAS clients >>> (https://twiki.cern.ch/twiki/bin/view/Atlas/XrootdVOMSsecPlugin). >>> This should be unrelated to your >>> issues, right? >>>> >>>> Ciao, >>>> Xavier. >>>> >>>>> -----Original Message----- >>>>> From: Lukasz Janyst [mailto:[log in to unmask]] >>>>> Sent: Friday, May 31, 2013 2:40 PM >>>>> To: Ilija Vukotic >>>>> Cc: Petzold, Andreas (SCC); Lukasz Janyst; >>>>> [log in to unmask]; Lincoln Bryant; Wei Yang; >>>>> dcache- >>>>> [log in to unmask] >>>>> Subject: [dcache-admin] Re: default ReadCacheSize value and other >>>>> questions >>>>> >>>>> Hi Andreas, >>>>> >>>>> The server behaves incorrectly and erratically when presented >>>>> with a >>>>> request for a chunk that exceeds the file boundary. Also, it sometimes >>>>> refuses to open a file that it has stated correctly. Please see the >>>>> attached test case. What kind of setup are you running on that box? Is >>>>> it a proxy, standard xrootd, something else? >>>>> >>>>> With the attached test program, I get the following on lxplus: >>>>> >>>>> ]==> ./a.out >>>>> File size: 797152257 >>>>> Read 16777216 bytes at offset 788529152 >>>>> Unable to read: [ERROR] Server responded with an error: [3011] >>>>> Unable to >>>>> read >>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>> >>>>> no such file or directory >>>>> >>>>> ]==> ./a.out >>>>> File size: 797152257 >>>>> Read 16777216 bytes at offset 788529152 >>>>> Status of all operations: OK >>>>> Received bytes: 8388608 >>>>> Should have received: 8623105 >>>>> >>>>> ]==> ./a.out >>>>> File size: 797152257 >>>>> Unable to open: [ERROR] Server responded with an error: [3005] >>>>> Unable to >>>>> open >>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>> >>>>> operation canceled >>>>> >>>>> In the case where read returned the OK response but incorrect >>>>> chunk, >>>>> xrdcopy should have been able to detect the error. I will make sure >>>>> it does. >>>>> >>>>> Cheers, >>>>> Lukasz >>>>> >>>>> >>>>> On 30.05.2013 20:35, Ilija Vukotic wrote: >>>>>> Hi Andreas, >>>>>> >>>>>> would you mind trying the same command from lxplus? >>>>>> >>>>>> Ilija >>>>>> >>>>>> -- >>>>>> Ilija Vukotic • Skype ivukotic • +1 872 230 6435 • University of >>>>>> Chicago >>>>>> >>>>>> >>>>>> >>>>>> On May 30, 2013, at 13:32 , Andreas Petzold <[log in to unmask]> >>>>>> wrote: >>>>>> >>>>>>> Hi Ilija, >>>>>>> >>>>>>> yes, in both cases I was using 3.3.2. >>>>>>> >>>>>>> Andreas >>>>>>> >>>>>>> >>>>>>> On 05/30/2013 08:25 PM, Ilija Vukotic wrote: >>>>>>>> Hi, >>>>>>>> Was the client 3.3.2? >>>>>>>> >>>>>>>> ilija >>>>>>>> >>>>>>>> Thumb typed >>>>>>>> >>>>>>>> ----- Reply message ----- >>>>>>>> From: "Andreas Petzold" <[log in to unmask]> >>>>>>>> To: "Ilija Vukotic" <[log in to unmask]> >>>>>>>> Cc: "Lukasz Janyst" <[log in to unmask]>, >>>>>>>> "[log in to unmask]" >>>>>>>> <[log in to unmask]>, "Lincoln Bryant" >>>>>>>> <[log in to unmask]>, "Wei Yang" <[log in to unmask]> >>>>>>>> Subject: default ReadCacheSize value and other questions >>>>>>>> Date: Thu, May 30, 2013 11:54 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi Ilja, >>>>>>>> >>>>>>>> FYI, I'm able to run xrdcopy successfully for your file on my >>>>>>>> desktop at >>>>>>>> KIT and on lxplus. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Andreas >>>>>>>> >>>>>>>> On 05/30/2013 06:04 PM, Ilija Vukotic wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I add Andreas and Guenter. Lincoln attempted to xrdcopy file >>>>>>>>> form FZK today between 17:00 and 18:00 (your time) and failed. >>>>> We would like to know what is in the servers log. Would you mind >>>>> looking it up? >>>>>>>>> (ordinary xrdcp works ok, but xrdcopy always fails from your >>>>>>>>> site.) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ilija >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On May 30, 2013, at 10:57 , Lukasz Janyst <[log in to unmask]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, this is a message that the client gets from the server. >>>>>>>>>> Can you actually check what is happening server-side? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Lukasz >>>>>>>>>> >>>>>>>>>> On 30.05.2013 17:55, Lincoln Bryant wrote: >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I also get this error. >>>>>>>>>>> >>>>>>>>>>> [lincolnb@uct2-int ~]$ xrdcopy --streams 1 root://f01-140-115- >>>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>>>> - > /dev/null >>>>>>>>>>> [ 98%][=================================================>] >>>>>>>>>>> [752M/760M] >>>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] Unable to >>>>>>>>>>> read >>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>> no such file or directory >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [lincolnb@uct2-int ~]$ rpm -q xrootd-client >>>>>>>>>>> xrootd-client-3.3.2-1.el5 >>>>>>>>>>> >>>>>>>>>>> --Lincoln >>>>>>>>>>> >>>>>>>>>>> On May 23, 2013, at 9:36 AM, Ilija Vukotic wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I added Wei and Lincoln to this mail so they can try too. >>>>>>>>>>>> >>>>>>>>>>>> I've just retried with both xrdcp and xrdcopy in a virgin >>>>>>>>>>>> shell on lxplus and still get the same behavior: >>>>>>>>>>>> >>>>>>>>>>>> ~ >xrdcopy --streams 1 root://f01-140-115- >>>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>>>> - > /dev/null >>>>>>>>>>>> [ 98%][=================================================>] >>>>>>>>>>>> [752M/760M] >>>>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] Unable >>>>>>>>>>>> to read >>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>> no such file or directory >>>>>>>>>>>> >>>>>>>>>>>> and >>>>>>>>>>>> >>>>>>>>>>>> xrdcp --streams 1 root://f01-140-115- >>>>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>>>> - > /dev/null >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Ilija >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Nullius in verba >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On May 23, 2013, at 2:07 , Lukasz Janyst <[log in to unmask]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I get this error with both clients. >>>>>>>>>>>>> >>>>>>>>>>>>> Lukasz >>>>>>>>>>>>> >>>>>>>>>>>>> On 23.05.2013 09:05, Lukasz Janyst wrote: >>>>>>>>>>>>>> Hi Ilija, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I constantly get: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ]==> xrdcopy --streams 1 >>>>>>>>>>>>>> root://f01-140-115- >>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>> >>>>>>>>>>>>>> - > /dev/null >>>>>>>>>>>>>> [100%][==================================================] >>>>>>>>>>>>>> [0/0] >>>>>>>>>>>>>> Run: [ERROR] Server responded with an error: [3005] Unable >>>>>>>>>>>>>> to open >>>>>>>>>>>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>>>>>>>>>>> >>>>>>>>>>>>>> operation canceled >>>>>>>>>>>>>> >>>>>>>>>>>>>> Are you sure the server config is OK? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Lukasz >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 23.05.2013 02:20, Ilija Vukotic wrote: >>>>>>>>>>>>>>> Hi Lukasz, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> when doing this: >>>>>>>>>>>>>>> xrdcopy --streams 1 >>>>>>>>>>>>>>> root://f01-140-115- >>> e.gridka.de:1094//atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root >>> >>>>>>>>>>>>>>> - > /dev/null >>>>>>>>>>>>>>> from lxplus it always (!) fails between 94% and 98% in >>>>>>>>>>>>>>> one of the two >>>>>>>>>>>>>>> ways. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [ 96%][================================================> >>>>>>>>>>>>>>> ] [736M/760M] >>>>>>>>>>>>>>> Run: [ERROR] Server responded with an error: [3011] >>>>>>>>>>>>>>> Unable to read >>>>>>>>>>>>>>> /atlas/dq2/user/ilijav/HCtest/user.ilijav.HCtest.1/group.test.hc.NTUP_SMWZ.root; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> no such file or directory >>>>>>>>>>>>>>> [ >>>>>>>>>>>>>>> 98%][=================================================>] >>>>>>>>>>>>>>> [748M/760M] >>>>>>>>>>>>>>> Run: [ERROR] Internal error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> at the same time standard xrdcp works as expected. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Security is off at that server so please feel free to try >>>>>>>>>>>>>>> and replicate. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Nullius in verba >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On May 22, 2013, at 10:47 , Ilija Vukotic >>>>>>>>>>>>>>> <[log in to unmask]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I was using standard xrdcp you get at lxplus (which is >>>>>>>>>>>>>>>> 3.3.2) >>>>>>>>>>>>>>>> Now I'll repeat it all with xrdcopy. Will let you know >>>>>>>>>>>>>>>> what I measure. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Nullius in verba >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On May 22, 2013, at 8:32 , Lukasz Janyst <[log in to unmask]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Andreas has now done some testing as well: >>>>>>>>>>>>>>>>> https://github.com/xrootd/xrootd/issues/20 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Lukasz >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 22.05.2013 14:34, Lukasz Janyst wrote: >>>>>>>>>>>>>>>>>> Hi Ilija, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> which version of xrootd are you using? Would it be >>>>>>>>>>>>>>>>>> possible for you >>>>>>>>>>>>>>>>>> to try the new client (xrdcopy) instead of the old one >>>>>>>>>>>>>>>>>> (xrdcp)? At >>>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>>> from version 3.3.2? We were able to get around 750MB/s >>>>>>>>>>>>>>>>>> with just one >>>>>>>>>>>>>>>>>> client on 10GB link. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The default for the old client is indeed one >>>>>>>>>>>>>>>>>> stream, but if you >>>>>>>>>>>>>>>>>> don't disable the cache completely it will send many >>>>>>>>>>>>>>>>>> requests >>>>>>>>>>>>>>>>>> synchronously. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> Lukasz >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 21.05.2013 09:00, Ilija Vukotic wrote: >>>>>>>>>>>>>>>>>>> Hi Lukasz, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I was trying to find out why xrdcp is not using the >>>>>>>>>>>>>>>>>>> full available >>>>>>>>>>>>>>>>>>> bandwidth at one 10Gb WAN link and so was playing >>>>>>>>>>>>>>>>>>> with parameters >>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>> number of streams, size of ReadCache etc. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> What I don't understand is that whatever value I set for >>>>>>>>>>>>>>>>>>> ReadCacheSize I >>>>>>>>>>>>>>>>>>> get performance roughly 10 times worse than if I set >>>>>>>>>>>>>>>>>>> nothing. >>>>>>>>>>>>>>>>>>> Strange thing here is that performance is 10x worse >>>>>>>>>>>>>>>>>>> even when I >>>>>>>>>>>>>>>>>>> set it >>>>>>>>>>>>>>>>>>> to it's default value: 4000000. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Also one more thing is puzzling to me: I have a way >>>>>>>>>>>>>>>>>>> to intercept >>>>>>>>>>>>>>>>>>> messages exchanged between xrootd server and client. >>>>>>>>>>>>>>>>>>> Looking in >>>>>>>>>>>>>>>>>>> them I >>>>>>>>>>>>>>>>>>> see that default behavior of xrdcp is to at the start >>>>>>>>>>>>>>>>>>> request 20 >>>>>>>>>>>>>>>>>>> chunks >>>>>>>>>>>>>>>>>>> of 4MB each. Documentation claims that the default is >>>>>>>>>>>>>>>>>>> one stream? >>>>>>>>>>>>>>>>>>> Could you explain this to me or point me to some >>>>>>>>>>>>>>>>>>> documentation I >>>>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>> read? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks a lot, >>>>>>>>>>>>>>>>>>> Ilija >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Dr Ilija Vukotic [log in to unmask] >>>>>>>>>>>>>>>>>>> University of Chicagohttp://www.vukotic.me >>>>>>>>>>>>>>>>>>> 5620 S Ellis Ave Tel: +1-773-702-7475 >>>>>>>>>>>>>>>>>>> Chicago IL 60637, USA >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Karlsruhe Institute of Technology (KIT) >>>>>>>> Steinbuch Centre for Computing (SCC) >>>>>>>> >>>>>>>> Andreas Petzold >>>>>>>> >>>>>>>> Hermann-von-Helmholtz-Platz 1, Building 449, Room 202 >>>>>>>> D-76344 Eggenstein-Leopoldshafen >>>>>>>> >>>>>>>> Tel: +49 721 608 24916 >>>>>>>> Fax: +49 721 608 24972 >>>>>>>> Email: [log in to unmask] >>>>>>>> www.scc.kit.edu <http://www.scc.kit.edu> >>>>>>>> >>>>>>>> KIT – University of the State of Baden-Wuerttemberg and >>>>>>>> National Research Center of the Helmholtz Association >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Karlsruhe Institute of Technology (KIT) >>>>>>> Steinbuch Centre for Computing (SCC) >>>>>>> >>>>>>> Andreas Petzold >>>>>>> >>>>>>> Hermann-von-Helmholtz-Platz 1, Building 449, Room 202 >>>>>>> D-76344 Eggenstein-Leopoldshafen >>>>>>> >>>>>>> Tel: +49 721 608 24916 >>>>>>> Fax: +49 721 608 24972 >>>>>>> Email: [log in to unmask] >>>>>>> www.scc.kit.edu >>>>>>> >>>>>>> KIT – University of the State of Baden-Wuerttemberg and >>>>>>> National Research Center of the Helmholtz Association >>>>>>> >>>>>> >>>> >> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1