Hi Adrian, Thanks for reporting this problem! I just pushed a patch that will allow for adding opaque info to local files: https://github.com/xrootd/xrootd/commit/45f4d8cf1cf2e7bdfc69c461dd47d45ae838748c with this fix you will be able to add the '?xrdcl.unzip=fn' cgi to the local metalink (I run some tests and it extracts the right file correctly). The patch will be included in 4.10.1 which we are now preparing. Cheers, Michal ________________________________________ From: Adrian Sevcenco Sent: 09 September 2019 12:18 To: Michal Kamil Simon; [log in to unmask] Subject: Re: python :: cp process fails when it shouldn't (another utility can download file) On 9/9/19 12:56 PM, Michal Kamil Simon wrote: > Hi Adrian, > > Could you try adding the /xrdcl.unzip=AliAOD.root/ cgi to the original > metalink URL, like: > > file://localhost/home/adrian/tmp/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root well, i do not think that the argument understand something else than a file : ll total 3266976 -rw-r--r-- 1 adrian adrian 3345375231 Sep 9 13:06 AliAOD.root.zip -rw-rw-r-- 1 adrian adrian 3289 Sep 9 13:02 _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4 xrdcp -p -P -f _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root AliAOD.root xrdcp: No such file or directory processing _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root xrdcp -p -P -f file://localhost/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root AliAOD.root xrdcp: No such file or directory processing /_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root xrdcp -p -P -f file://localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root AliAOD.root xrdcp: No such file or directory processing //home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root xrdcp -p -P -f file:///localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root AliAOD.root xrdcp: No such file or directory processing /localhost//home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root xrdcp -p -P -f file://${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root AliAOD.root xrdcp: No such file or directory processing /home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root Thank you! Adrian > > (it seems that there's a friction between the metalink handling and zip > archive handling) > > Cheers, > Michal > ________________________________________ > From: Adrian Sevcenco > Sent: 06 September 2019 16:30 > To: Michal Kamil Simon; [log in to unmask] > Subject: Re: python :: cp process fails when it shouldn't (another > utility can download file) > > Hi Michal! > > On 9/2/19 5:52 PM, Michal Kamil Simon wrote: > > Yep, it looks about right :-) > so, i use ?xrdcl.unzip=AliAOD.root&authz=TOKEN > format but it seems that the zipfile is downloaded instead of file > extracted from archive... > > the detailed log is here: > https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP > > do you have any idea/hint why the file is not extracted? > > Thanks a lot!! > Adrian > > > > > > > Michal > > ________________________________________ > > From: Adrian Sevcenco > > Sent: 02 September 2019 16:46 > > To: Michal Kamil Simon; [log in to unmask] > > Subject: Re: python :: cp process fails when it shouldn't (another > utility can download file) > > > > On 9/2/19 5:29 PM, Michal Kamil Simon wrote: > >> Hi Adrian, > > Hi! > > > >> You can replace the '#' with '?xrdcl.unzip=' however you have to make > >> sure that if the URL > >> already contains a CGI you have to replace the following '?' with '&', > >> e.g. : > >> > >> > root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*#*AliAOD.root*?*authz=LONG_ALICE_TOKEN > >> > >> gets transformed into: > >> > >> > root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*?xrdcl.unzip=*AliAOD.root*&*authz=LONG_ALICE_TOKEN > > > > yeap, is doable as the full url (physical url + token) is constructed by > > me... so i can do something like : > > > > pfn_components = pfn.split('#') # i have the guarantee that the files > > that ALICE uploads have no # in name > > > > if len(pfn_components) > 1: > > full_url = pfn + '?xrdcl.unzip=' + pfn_components[1] + > > '&authz=LONG_ALICE_TOKEN' > > else: > > full_url = pfn + '?authz=LONG_ALICE_TOKEN' > > > > Does it sound right? > > Thanks a lot for help!! > > Adrian > > > > > > > >> > >> > >> > >> Regarding the /CopyProcess.add_job(...)/ method I could add parameters > >> that will allow > >> to specify the file name for extraction from zip archive. > >> > >> Regarding supporting the '#' root native format we will have to see with > >> Andy whether this > >> wont harm any existing use cases (as # is a legal character that could > >> be used a file name). > >> > >> Cheers, > >> Michal > >> ________________________________________ > >> From: Adrian Sevcenco > >> Sent: 02 September 2019 15:37 > >> To: Michal Kamil Simon; [log in to unmask] > >> Subject: Re: python :: cp process fails when it shouldn't (another > >> utility can download file) > >> > >> On 9/2/19 2:09 PM, Michal Kamil Simon wrote: > >> > Hi Adrian, > >> Hi! > >> > >> > >From what I see in the logs you use the following file name: > >> > > >> > > >> > root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66#AliAOD.root > >> > > >> > The '#' is root syntax for unpacking root files, this is not supported > >> > in the > >> > xrootd client, instead you have to use the /xrdcl.unzip/ cgi tag, e.g. > >> > > >> > > >> > root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66?xrdcl.unzip=AliAOD.root > >> > >> oh!!! so, could i use a simplistic logic like : > >> replace latest '#' from string with '?xrdcl.unzip=' > >> > >> ALICE stores files in the form of GUID (that last uid) > >> and when i request access to a lfn i get the guid and the authz envelope > >> for accessing the file ... so, it is guaranteed that i will always get a > >> url with a GUID ... > >> > >> Given this, do you thing that i could use the logic from above? > >> > >> > alternatively I can expose extracting of zip files (root files use zip > >> > format for bundling) > >> > in the /CopyProcess.add_job(...)/ method. > >> that would be great! if it is possible it would be best if > >> the same format of '#file' is recognized (as this is the url that i get > >> when requesting lfn access) > >> > >> Thanks a lot!! > >> Adrian > >> > >> > > >> > Hope this helps! > >> > > >> > Cheers, > >> > Michal > >> > > >> > ________________________________________ > >> > From: Adrian Sevcenco > >> > Sent: 01 September 2019 22:13 > >> > To: [log in to unmask] > >> > Cc: Michal Kamil Simon > >> > Subject: python :: cp process fails when it shouldn't (another utility > >> > can download file) > >> > > >> > Hi! I have a really baffling situation where my python tool cannot > >> > download a file and another tool (java based, use xrdcp) can download > >> > the same file ... > >> > > >> > the detailed logs for my cp are here : > >> > https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP > >> > > >> > the java based tool it seems that somehow ignores the external XRD_ > >> > variables so i cannot get a log of cp process > >> > > >> > Could some expert take a look please and point me to a hint why my cp > >> > fails and the other tool can download just fine? > >> > > >> > Thank you!! > >> > Adrian > >> > > >> > >> > >> -- > >> ---------------------------------------------- > >> Adrian Sevcenco, Ph.D. | > >> Institute of Space Science - ISS, Romania | > >> adrian.sevcenco at {cern.ch,spacescience.ro} | > >> ---------------------------------------------- > >> > > > > > > -- > > ---------------------------------------------- > > Adrian Sevcenco, Ph.D. | > > Institute of Space Science - ISS, Romania | > > adrian.sevcenco at {cern.ch,spacescience.ro} | > > ---------------------------------------------- > > > > > > > > > -- > ---------------------------------------------- > Adrian Sevcenco, Ph.D. | > Institute of Space Science - ISS, Romania | > adrian.sevcenco at {cern.ch,spacescience.ro} | > ---------------------------------------------- > -- ---------------------------------------------- Adrian Sevcenco, Ph.D. | Institute of Space Science - ISS, Romania | adrian.sevcenco at {cern.ch,spacescience.ro} | ---------------------------------------------- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1