Hi Adrian, Sorry for the late response, to answer your question no they binaries are not required, I just pushed a patch, they are not installed any more: https://github.com/xrootd/xrootd/commit/4974edaa10f23ec94e4877c8747f7518a5753f50 Checking the libs installed by rpms doesn't make sense, as the user may decided at any point to upgrade/downgrade/remove them making the bindings unusable. Cheers, Michal ________________________________________ From: Adrian Sevcenco Sent: 11 September 2019 15:38 To: Michal Kamil Simon; [log in to unmask] Subject: Re: python :: cp process fails when it shouldn't (another utility can download file) On 9/11/19 4:16 PM, Michal Kamil Simon wrote: > Hi Adrian, Hi! > Yep decompression is feasible, though I'm not sure I'm able to implement > it early enough so it gets included in the next release. Let's see ;-) it does not matter :) later will be always better than never :)) > Please create a feature request in our github ;-) done :) https://github.com/xrootd/xrootd/issues/1054 btw, i was not sure if it is worth to start another mail thread or issue: i notices that the compilation of xrootd client part for python installation also creates a lot of executable binaries in .local/lib/python3.7/site-packages/xrdcl/bin 1. are these required? maybe a flag could be added so the building/linking of these does not happen? one reason would be that many defaults have .local/bin in PATH so these could have a not healthy interaction with system binaries and libs 2. given that a set of xrootd libs could be already present in system, is there a way to check compatibility with the system libs and no longer build/compile the python oriented client libs? Thank you!! Adrian > > Cheers, > Michal > ________________________________________ > From: Adrian Sevcenco > Sent: 11 September 2019 15:06 > To: Michal Kamil Simon; [log in to unmask] > Subject: Re: python :: cp process fails when it shouldn't (another utility can download file) > > On 9/11/19 12:15 PM, Michal Kamil Simon wrote: >> Hi Adrian, > Hi! > >> Your ZIP file you use for testing is compressed, for now we only support >> extraction from ZIP files >> that are not compressed (our use case are ROOT files that use ZIP format >> only for bundling). > oh... > >> You can create an uncompressed ZIP file for testing using following command: >> *zip -0 foo.zip /path/to/files/foo* > all ALICE data and MC central productions are stored in compressed zip > files.. so it does not matter the use case of uncompressed zips > > Is there any chance to have also the extraction from compressed zips? > > In my naive thinking is that if you use anyway vector reads to read the > central directory record and then the file entries (local file header, > encryption file header, file data and data descriptor) you can use the > info from local file header to decompress the data > (from what i see at first glance in > https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) > > Or should i go towards downloading the zip file to temporary and extract > the needed files on the client? > > Thanks a lot!! > Adrian > > > >> >> Cheers, >> Michal >> ________________________________________ >> From: Adrian Sevcenco >> Sent: 11 September 2019 09:59 >> To: Michal Kamil Simon; [log in to unmask] >> Subject: Re: python :: cp process fails when it shouldn't (another >> utility can download file) >> >> On 9/10/19 2:27 PM, Michal Kamil Simon wrote: >> > Hi Adrian, >> Hi Michal! >> >> > Yes, this should also work from python. >> > >> > You just need to add the ?xrdcl.unzip=fn to the metalink url (nothing >> else), also remove the #fn from the files >> > inside the metalink. >> well, it is not working and i get this kind of messages: >> jobID: 1/1 >>> STATUS: 1 ; ERRNO: 0 ; CODE: 13 ; MESSAGE: [ERROR] >> Operation not supported: Decompression is not supported! >> >> full xrootd log, and my cli debug output (and metalink) are here: >> https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP >> >> the actual metalink was named with % but it seems that EOS have a bug >> and deletes the files with % in names (i cannot copy through webdav the >> original meta file) >> >> Thanks a lot!! >> Adrian >> >> >> > >> > To make the xrdcl.unzip work for the links/files inside metalink is >> more complicated, if this is something >> > you would be interested in, please create an issue in github, but I >> don't think we will be able to accommodate >> > this feature request for 4.10.1. >> > >> > It is the client that is extracting the file from the zip archive - >> basically the client reads the central directory record >> > of the zip archives in order to know the offset of the respective >> file within the archive. >> > >> > Cheers, >> > Michal >> > ________________________________________ >> > From: Adrian Sevcenco >> > Sent: 10 September 2019 13:11 >> > To: Michal Kamil Simon; [log in to unmask] >> > Subject: Re: python :: cp process fails when it shouldn't (another >> utility can download file) >> > >> > On 9/10/19 1:22 PM, Michal Kamil Simon wrote: >> >> Hi Adrian, >> > Hi! >> > >> >> Thanks for reporting this problem! >> >> >> >> I just pushed a patch that will allow for adding opaque info to >> local files: >> >> >> https://github.com/xrootd/xrootd/commit/45f4d8cf1cf2e7bdfc69c461dd47d45ae838748c >> > great! thanks a lot!!! >> > >> >> with this fix you will be able to add the '?xrdcl.unzip=fn' cgi to >> the local metalink (I run some tests >> >> and it extracts the right file correctly). >> > So, would this metfile.meta4?xrdcl.unzip=zipped_file work for the python >> > copyProcess? >> > >> > Also, would these parameters be added? (the url from metalink have the >> > ALICE authorization envelope) >> > >> >> The patch will be included in 4.10.1 which we are now preparing. >> > any chance of adding this feature also to the links from within the >> > metalink? i do not know what is happening within, so i just ask :) >> > >> > If the unziping is done on the server side, maybe it would make more >> > sense to work with actual url (from metalink), the way that you >> > initially told me to do? >> > >> > Thanks a lot! >> > Adrian >> > >> > >> > >> >> >> >> Cheers, >> >> Michal >> >> ________________________________________ >> >> From: Adrian Sevcenco >> >> Sent: 09 September 2019 12:18 >> >> To: Michal Kamil Simon; [log in to unmask] >> >> Subject: Re: python :: cp process fails when it shouldn't (another >> utility can download file) >> >> >> >> On 9/9/19 12:56 PM, Michal Kamil Simon wrote: >> >>> Hi Adrian, >> >>> >> >>> Could you try adding the /xrdcl.unzip=AliAOD.root/ cgi to the original >> >>> metalink URL, like: >> >>> >> >>> >> file://localhost/home/adrian/tmp/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> well, i do not think that the argument understand something else than a >> >> file : >> >> >> >> ll >> >> total 3266976 >> >> -rw-r--r-- 1 adrian adrian 3345375231 Sep 9 13:06 AliAOD.root.zip >> >> -rw-rw-r-- 1 adrian adrian 3289 Sep 9 13:02 >> >> >> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4 >> >> >> >> xrdcp -p -P -f >> >> >> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> AliAOD.root >> >> xrdcp: No such file or directory processing >> >> >> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> >> >> xrdcp -p -P -f >> >> >> file://localhost/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> AliAOD.root >> >> xrdcp: No such file or directory processing >> >> >> /_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> >> >> xrdcp -p -P -f >> >> >> file://localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> AliAOD.root >> >> xrdcp: No such file or directory processing >> >> >> //home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> >> >> xrdcp -p -P -f >> >> >> file:///localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> AliAOD.root >> >> xrdcp: No such file or directory processing >> >> >> /localhost//home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> >> >> xrdcp -p -P -f >> >> >> file://${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> AliAOD.root >> >> xrdcp: No such file or directory processing >> >> >> /home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root >> >> >> >> Thank you! >> >> Adrian >> >> >> >>> >> >>> (it seems that there's a friction between the metalink handling and zip >> >>> archive handling) >> >>> >> >>> Cheers, >> >>> Michal >> >>> ________________________________________ >> >>> From: Adrian Sevcenco >> >>> Sent: 06 September 2019 16:30 >> >>> To: Michal Kamil Simon; [log in to unmask] >> >>> Subject: Re: python :: cp process fails when it shouldn't (another >> >>> utility can download file) >> >>> >> >>> Hi Michal! >> >>> >> >>> On 9/2/19 5:52 PM, Michal Kamil Simon wrote: >> >>> > Yep, it looks about right :-) >> >>> so, i use ?xrdcl.unzip=AliAOD.root&authz=TOKEN >> >>> format but it seems that the zipfile is downloaded instead of file >> >>> extracted from archive... >> >>> >> >>> the detailed log is here: >> >>> https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP >> >>> >> >>> do you have any idea/hint why the file is not extracted? >> >>> >> >>> Thanks a lot!! >> >>> Adrian >> >>> >> >>> >> >>> >> >>> > >> >>> > Michal >> >>> > ________________________________________ >> >>> > From: Adrian Sevcenco >> >>> > Sent: 02 September 2019 16:46 >> >>> > To: Michal Kamil Simon; [log in to unmask] >> >>> > Subject: Re: python :: cp process fails when it shouldn't (another >> >>> utility can download file) >> >>> > >> >>> > On 9/2/19 5:29 PM, Michal Kamil Simon wrote: >> >>> >> Hi Adrian, >> >>> > Hi! >> >>> > >> >>> >> You can replace the '#' with '?xrdcl.unzip=' however you have to >> make >> >>> >> sure that if the URL >> >>> >> already contains a CGI you have to replace the following '?' >> with '&', >> >>> >> e.g. : >> >>> >> >> >>> >> >> >>> >> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*#*AliAOD.root*?*authz=LONG_ALICE_TOKEN >> >>> >> >> >>> >> gets transformed into: >> >>> >> >> >>> >> >> >>> >> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*?xrdcl.unzip=*AliAOD.root*&*authz=LONG_ALICE_TOKEN >> >>> > >> >>> > yeap, is doable as the full url (physical url + token) is >> constructed by >> >>> > me... so i can do something like : >> >>> > >> >>> > pfn_components = pfn.split('#') # i have the guarantee that the files >> >>> > that ALICE uploads have no # in name >> >>> > >> >>> > if len(pfn_components) > 1: >> >>> > full_url = pfn + '?xrdcl.unzip=' + pfn_components[1] + >> >>> > '&authz=LONG_ALICE_TOKEN' >> >>> > else: >> >>> > full_url = pfn + '?authz=LONG_ALICE_TOKEN' >> >>> > >> >>> > Does it sound right? >> >>> > Thanks a lot for help!! >> >>> > Adrian >> >>> > >> >>> > >> >>> > >> >>> >> >> >>> >> >> >>> >> >> >>> >> Regarding the /CopyProcess.add_job(...)/ method I could add >> parameters >> >>> >> that will allow >> >>> >> to specify the file name for extraction from zip archive. >> >>> >> >> >>> >> Regarding supporting the '#' root native format we will have to >> see with >> >>> >> Andy whether this >> >>> >> wont harm any existing use cases (as # is a legal character that >> could >> >>> >> be used a file name). >> >>> >> >> >>> >> Cheers, >> >>> >> Michal >> >>> >> ________________________________________ >> >>> >> From: Adrian Sevcenco >> >>> >> Sent: 02 September 2019 15:37 >> >>> >> To: Michal Kamil Simon; [log in to unmask] >> >>> >> Subject: Re: python :: cp process fails when it shouldn't (another >> >>> >> utility can download file) >> >>> >> >> >>> >> On 9/2/19 2:09 PM, Michal Kamil Simon wrote: >> >>> >> > Hi Adrian, >> >>> >> Hi! >> >>> >> >> >>> >> > >From what I see in the logs you use the following file name: >> >>> >> > >> >>> >> > >> >>> >> >> >>> >> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66#AliAOD.root >> >>> >> > >> >>> >> > The '#' is root syntax for unpacking root files, this is not >> supported >> >>> >> > in the >> >>> >> > xrootd client, instead you have to use the /xrdcl.unzip/ cgi >> tag, e.g. >> >>> >> > >> >>> >> > >> >>> >> >> >>> >> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66?xrdcl.unzip=AliAOD.root >> >>> >> >> >>> >> oh!!! so, could i use a simplistic logic like : >> >>> >> replace latest '#' from string with '?xrdcl.unzip=' >> >>> >> >> >>> >> ALICE stores files in the form of GUID (that last uid) >> >>> >> and when i request access to a lfn i get the guid and the authz >> envelope >> >>> >> for accessing the file ... so, it is guaranteed that i will >> always get a >> >>> >> url with a GUID ... >> >>> >> >> >>> >> Given this, do you thing that i could use the logic from above? >> >>> >> >> >>> >> > alternatively I can expose extracting of zip files (root files >> use zip >> >>> >> > format for bundling) >> >>> >> > in the /CopyProcess.add_job(...)/ method. >> >>> >> that would be great! if it is possible it would be best if >> >>> >> the same format of '#file' is recognized (as this is the url >> that i get >> >>> >> when requesting lfn access) >> >>> >> >> >>> >> Thanks a lot!! >> >>> >> Adrian >> >>> >> >> >>> >> > >> >>> >> > Hope this helps! >> >>> >> > >> >>> >> > Cheers, >> >>> >> > Michal >> >>> >> > >> >>> >> > ________________________________________ >> >>> >> > From: Adrian Sevcenco >> >>> >> > Sent: 01 September 2019 22:13 >> >>> >> > To: [log in to unmask] >> >>> >> > Cc: Michal Kamil Simon >> >>> >> > Subject: python :: cp process fails when it shouldn't (another >> utility >> >>> >> > can download file) >> >>> >> > >> >>> >> > Hi! I have a really baffling situation where my python tool cannot >> >>> >> > download a file and another tool (java based, use xrdcp) can >> download >> >>> >> > the same file ... >> >>> >> > >> >>> >> > the detailed logs for my cp are here : >> >>> >> > https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP >> >>> >> > >> >>> >> > the java based tool it seems that somehow ignores the external >> XRD_ >> >>> >> > variables so i cannot get a log of cp process >> >>> >> > >> >>> >> > Could some expert take a look please and point me to a hint >> why my cp >> >>> >> > fails and the other tool can download just fine? >> >>> >> > >> >>> >> > Thank you!! >> >>> >> > Adrian >> >>> >> > >> > > > -- > ---------------------------------------------- > Adrian Sevcenco, Ph.D. | > Institute of Space Science - ISS, Romania | > adrian.sevcenco at {cern.ch,spacescience.ro} | > ---------------------------------------------- > > -- ---------------------------------------------- Adrian Sevcenco, Ph.D. | Institute of Space Science - ISS, Romania | adrian.sevcenco at {cern.ch,spacescience.ro} | ---------------------------------------------- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1