On 9/16/19 4:23 PM, Michal Kamil Simon wrote:
> Hi Adrian,
Hi!
> Sorry for the late response, to answer your question no they binaries
no problem
> are not required,
> I just pushed a patch, they are not installed any more:
> https://github.com/xrootd/xrootd/commit/4974edaa10f23ec94e4877c8747f7518a5753f50
yeap, confirmed.
well, my other though was if would have been possible to disable their
linking/generation to have a reduced build time ... but it's mostly
irrelevant
> Checking the libs installed by rpms doesn't make sense, as the user
> may decided at any point to upgrade/downgrade/remove them making
> the bindings unusable.
true ... and if all xrootd is built by hand, both python bindings and
libs will be created anyway
And just to have it also here, i can confirm that i could get a file
from a compressed zip
Thank you very much!!
Adrian
>
> Cheers,
> Michal
> ________________________________________
> From: Adrian Sevcenco
> Sent: 11 September 2019 15:38
> To: Michal Kamil Simon; [log in to unmask]
> Subject: Re: python :: cp process fails when it shouldn't (another
> utility can download file)
>
> On 9/11/19 4:16 PM, Michal Kamil Simon wrote:
> > Hi Adrian,
> Hi!
>
> > Yep decompression is feasible, though I'm not sure I'm able to implement
> > it early enough so it gets included in the next release. Let's see ;-)
> it does not matter :) later will be always better than never :))
>
> > Please create a feature request in our github ;-)
> done :) https://github.com/xrootd/xrootd/issues/1054
>
> btw, i was not sure if it is worth to start another mail thread or
> issue: i notices that the compilation of xrootd client part for python
> installation also creates a lot of executable binaries in
> .local/lib/python3.7/site-packages/xrdcl/bin
>
> 1. are these required? maybe a flag could be added so the
> building/linking of these does not happen?
> one reason would be that many defaults have .local/bin in PATH so these
> could have a not healthy interaction with system binaries and libs
>
> 2. given that a set of xrootd libs could be already present in system,
> is there a way to check compatibility with the system libs and no longer
> build/compile the python oriented client libs?
>
> Thank you!!
> Adrian
>
>
> >
> > Cheers,
> > Michal
> > ________________________________________
> > From: Adrian Sevcenco
> > Sent: 11 September 2019 15:06
> > To: Michal Kamil Simon; [log in to unmask]
> > Subject: Re: python :: cp process fails when it shouldn't (another
> utility can download file)
> >
> > On 9/11/19 12:15 PM, Michal Kamil Simon wrote:
> >> Hi Adrian,
> > Hi!
> >
> >> Your ZIP file you use for testing is compressed, for now we only support
> >> extraction from ZIP files
> >> that are not compressed (our use case are ROOT files that use ZIP format
> >> only for bundling).
> > oh...
> >
> >> You can create an uncompressed ZIP file for testing using following
> command:
> >> *zip -0 foo.zip /path/to/files/foo*
> > all ALICE data and MC central productions are stored in compressed zip
> > files.. so it does not matter the use case of uncompressed zips
> >
> > Is there any chance to have also the extraction from compressed zips?
> >
> > In my naive thinking is that if you use anyway vector reads to read the
> > central directory record and then the file entries (local file header,
> > encryption file header, file data and data descriptor) you can use the
> > info from local file header to decompress the data
> > (from what i see at first glance in
> > https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT)
> >
> > Or should i go towards downloading the zip file to temporary and extract
> > the needed files on the client?
> >
> > Thanks a lot!!
> > Adrian
> >
> >
> >
> >>
> >> Cheers,
> >> Michal
> >> ________________________________________
> >> From: Adrian Sevcenco
> >> Sent: 11 September 2019 09:59
> >> To: Michal Kamil Simon; [log in to unmask]
> >> Subject: Re: python :: cp process fails when it shouldn't (another
> >> utility can download file)
> >>
> >> On 9/10/19 2:27 PM, Michal Kamil Simon wrote:
> >> > Hi Adrian,
> >> Hi Michal!
> >>
> >> > Yes, this should also work from python.
> >> >
> >> > You just need to add the ?xrdcl.unzip=fn to the metalink url (nothing
> >> else), also remove the #fn from the files
> >> > inside the metalink.
> >> well, it is not working and i get this kind of messages:
> >> jobID: 1/1 >>> STATUS: 1 ; ERRNO: 0 ; CODE: 13 ; MESSAGE: [ERROR]
> >> Operation not supported: Decompression is not supported!
> >>
> >> full xrootd log, and my cli debug output (and metalink) are here:
> >> https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP
> >>
> >> the actual metalink was named with % but it seems that EOS have a bug
> >> and deletes the files with % in names (i cannot copy through webdav the
> >> original meta file)
> >>
> >> Thanks a lot!!
> >> Adrian
> >>
> >>
> >> >
> >> > To make the xrdcl.unzip work for the links/files inside metalink is
> >> more complicated, if this is something
> >> > you would be interested in, please create an issue in github, but I
> >> don't think we will be able to accommodate
> >> > this feature request for 4.10.1.
> >> >
> >> > It is the client that is extracting the file from the zip archive -
> >> basically the client reads the central directory record
> >> > of the zip archives in order to know the offset of the respective
> >> file within the archive.
> >> >
> >> > Cheers,
> >> > Michal
> >> > ________________________________________
> >> > From: Adrian Sevcenco
> >> > Sent: 10 September 2019 13:11
> >> > To: Michal Kamil Simon; [log in to unmask]
> >> > Subject: Re: python :: cp process fails when it shouldn't (another
> >> utility can download file)
> >> >
> >> > On 9/10/19 1:22 PM, Michal Kamil Simon wrote:
> >> >> Hi Adrian,
> >> > Hi!
> >> >
> >> >> Thanks for reporting this problem!
> >> >>
> >> >> I just pushed a patch that will allow for adding opaque info to
> >> local files:
> >> >>
> >>
> https://github.com/xrootd/xrootd/commit/45f4d8cf1cf2e7bdfc69c461dd47d45ae838748c
> >> > great! thanks a lot!!!
> >> >
> >> >> with this fix you will be able to add the '?xrdcl.unzip=fn' cgi to
> >> the local metalink (I run some tests
> >> >> and it extracts the right file correctly).
> >> > So, would this metfile.meta4?xrdcl.unzip=zipped_file work for the
> python
> >> > copyProcess?
> >> >
> >> > Also, would these parameters be added? (the url from metalink have the
> >> > ALICE authorization envelope)
> >> >
> >> >> The patch will be included in 4.10.1 which we are now preparing.
> >> > any chance of adding this feature also to the links from within the
> >> > metalink? i do not know what is happening within, so i just ask :)
> >> >
> >> > If the unziping is done on the server side, maybe it would make more
> >> > sense to work with actual url (from metalink), the way that you
> >> > initially told me to do?
> >> >
> >> > Thanks a lot!
> >> > Adrian
> >> >
> >> >
> >> >
> >> >>
> >> >> Cheers,
> >> >> Michal
> >> >> ________________________________________
> >> >> From: Adrian Sevcenco
> >> >> Sent: 09 September 2019 12:18
> >> >> To: Michal Kamil Simon; [log in to unmask]
> >> >> Subject: Re: python :: cp process fails when it shouldn't (another
> >> utility can download file)
> >> >>
> >> >> On 9/9/19 12:56 PM, Michal Kamil Simon wrote:
> >> >>> Hi Adrian,
> >> >>>
> >> >>> Could you try adding the /xrdcl.unzip=AliAOD.root/ cgi to the
> original
> >> >>> metalink URL, like:
> >> >>>
> >> >>>
> >>
> file://localhost/home/adrian/tmp/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> well, i do not think that the argument understand something else
> than a
> >> >> file :
> >> >>
> >> >> ll
> >> >> total 3266976
> >> >> -rw-r--r-- 1 adrian adrian 3345375231 Sep 9 13:06 AliAOD.root.zip
> >> >> -rw-rw-r-- 1 adrian adrian 3289 Sep 9 13:02
> >> >>
> >>
> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4
> >> >>
> >> >> xrdcp -p -P -f
> >> >>
> >>
> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> AliAOD.root
> >> >> xrdcp: No such file or directory processing
> >> >>
> >>
> _alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >>
> >> >> xrdcp -p -P -f
> >> >>
> >>
> file://localhost/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> AliAOD.root
> >> >> xrdcp: No such file or directory processing
> >> >>
> >>
> /_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >>
> >> >> xrdcp -p -P -f
> >> >>
> >>
> file://localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> AliAOD.root
> >> >> xrdcp: No such file or directory processing
> >> >>
> >>
> //home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >>
> >> >> xrdcp -p -P -f
> >> >>
> >>
> file:///localhost/${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> AliAOD.root
> >> >> xrdcp: No such file or directory processing
> >> >>
> >>
> /localhost//home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >>
> >> >> xrdcp -p -P -f
> >> >>
> >>
> file://${PWD}/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >> AliAOD.root
> >> >> xrdcp: No such file or directory processing
> >> >>
> >>
> /home/adrian/work-GRID/jalien_py/t/_alice_data_2018_LHC18m_000291397_pass1_withTRDtracking_AOD208_0884_AliAOD.root.meta4?xrdcl.unzip=AliAOD.root
> >> >>
> >> >> Thank you!
> >> >> Adrian
> >> >>
> >> >>>
> >> >>> (it seems that there's a friction between the metalink handling
> and zip
> >> >>> archive handling)
> >> >>>
> >> >>> Cheers,
> >> >>> Michal
> >> >>> ________________________________________
> >> >>> From: Adrian Sevcenco
> >> >>> Sent: 06 September 2019 16:30
> >> >>> To: Michal Kamil Simon; [log in to unmask]
> >> >>> Subject: Re: python :: cp process fails when it shouldn't (another
> >> >>> utility can download file)
> >> >>>
> >> >>> Hi Michal!
> >> >>>
> >> >>> On 9/2/19 5:52 PM, Michal Kamil Simon wrote:
> >> >>> > Yep, it looks about right :-)
> >> >>> so, i use ?xrdcl.unzip=AliAOD.root&authz=TOKEN
> >> >>> format but it seems that the zipfile is downloaded instead of file
> >> >>> extracted from archive...
> >> >>>
> >> >>> the detailed log is here:
> >> >>> https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP
> >> >>>
> >> >>> do you have any idea/hint why the file is not extracted?
> >> >>>
> >> >>> Thanks a lot!!
> >> >>> Adrian
> >> >>>
> >> >>>
> >> >>>
> >> >>> >
> >> >>> > Michal
> >> >>> > ________________________________________
> >> >>> > From: Adrian Sevcenco
> >> >>> > Sent: 02 September 2019 16:46
> >> >>> > To: Michal Kamil Simon; [log in to unmask]
> >> >>> > Subject: Re: python :: cp process fails when it shouldn't (another
> >> >>> utility can download file)
> >> >>> >
> >> >>> > On 9/2/19 5:29 PM, Michal Kamil Simon wrote:
> >> >>> >> Hi Adrian,
> >> >>> > Hi!
> >> >>> >
> >> >>> >> You can replace the '#' with '?xrdcl.unzip=' however you have to
> >> make
> >> >>> >> sure that if the URL
> >> >>> >> already contains a CGI you have to replace the following '?'
> >> with '&',
> >> >>> >> e.g. :
> >> >>> >>
> >> >>> >>
> >> >>>
> >>
> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*#*AliAOD.root*?*authz=LONG_ALICE_TOKEN
> >> >>> >>
> >> >>> >> gets transformed into:
> >> >>> >>
> >> >>> >>
> >> >>>
> >>
> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66*?xrdcl.unzip=*AliAOD.root*&*authz=LONG_ALICE_TOKEN
> >> >>> >
> >> >>> > yeap, is doable as the full url (physical url + token) is
> >> constructed by
> >> >>> > me... so i can do something like :
> >> >>> >
> >> >>> > pfn_components = pfn.split('#') # i have the guarantee that
> the files
> >> >>> > that ALICE uploads have no # in name
> >> >>> >
> >> >>> > if len(pfn_components) > 1:
> >> >>> > full_url = pfn + '?xrdcl.unzip=' + pfn_components[1] +
> >> >>> > '&authz=LONG_ALICE_TOKEN'
> >> >>> > else:
> >> >>> > full_url = pfn + '?authz=LONG_ALICE_TOKEN'
> >> >>> >
> >> >>> > Does it sound right?
> >> >>> > Thanks a lot for help!!
> >> >>> > Adrian
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> Regarding the /CopyProcess.add_job(...)/ method I could add
> >> parameters
> >> >>> >> that will allow
> >> >>> >> to specify the file name for extraction from zip archive.
> >> >>> >>
> >> >>> >> Regarding supporting the '#' root native format we will have to
> >> see with
> >> >>> >> Andy whether this
> >> >>> >> wont harm any existing use cases (as # is a legal character that
> >> could
> >> >>> >> be used a file name).
> >> >>> >>
> >> >>> >> Cheers,
> >> >>> >> Michal
> >> >>> >> ________________________________________
> >> >>> >> From: Adrian Sevcenco
> >> >>> >> Sent: 02 September 2019 15:37
> >> >>> >> To: Michal Kamil Simon; [log in to unmask]
> >> >>> >> Subject: Re: python :: cp process fails when it shouldn't
> (another
> >> >>> >> utility can download file)
> >> >>> >>
> >> >>> >> On 9/2/19 2:09 PM, Michal Kamil Simon wrote:
> >> >>> >> > Hi Adrian,
> >> >>> >> Hi!
> >> >>> >>
> >> >>> >> > >From what I see in the logs you use the following file name:
> >> >>> >> >
> >> >>> >> >
> >> >>> >>
> >> >>>
> >>
> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66#AliAOD.root
> >> >>> >> >
> >> >>> >> > The '#' is root syntax for unpacking root files, this is not
> >> supported
> >> >>> >> > in the
> >> >>> >> > xrootd client, instead you have to use the /xrdcl.unzip/ cgi
> >> tag, e.g.
> >> >>> >> >
> >> >>> >> >
> >> >>> >>
> >> >>>
> >>
> root://eosalice.cern.ch:1094//15/62933/56da6906-9149-11e7-ba1b-579516ed5c66?xrdcl.unzip=AliAOD.root
> >> >>> >>
> >> >>> >> oh!!! so, could i use a simplistic logic like :
> >> >>> >> replace latest '#' from string with '?xrdcl.unzip='
> >> >>> >>
> >> >>> >> ALICE stores files in the form of GUID (that last uid)
> >> >>> >> and when i request access to a lfn i get the guid and the authz
> >> envelope
> >> >>> >> for accessing the file ... so, it is guaranteed that i will
> >> always get a
> >> >>> >> url with a GUID ...
> >> >>> >>
> >> >>> >> Given this, do you thing that i could use the logic from above?
> >> >>> >>
> >> >>> >> > alternatively I can expose extracting of zip files (root files
> >> use zip
> >> >>> >> > format for bundling)
> >> >>> >> > in the /CopyProcess.add_job(...)/ method.
> >> >>> >> that would be great! if it is possible it would be best if
> >> >>> >> the same format of '#file' is recognized (as this is the url
> >> that i get
> >> >>> >> when requesting lfn access)
> >> >>> >>
> >> >>> >> Thanks a lot!!
> >> >>> >> Adrian
> >> >>> >>
> >> >>> >> >
> >> >>> >> > Hope this helps!
> >> >>> >> >
> >> >>> >> > Cheers,
> >> >>> >> > Michal
> >> >>> >> >
> >> >>> >> > ________________________________________
> >> >>> >> > From: Adrian Sevcenco
> >> >>> >> > Sent: 01 September 2019 22:13
> >> >>> >> > To: [log in to unmask]
> >> >>> >> > Cc: Michal Kamil Simon
> >> >>> >> > Subject: python :: cp process fails when it shouldn't (another
> >> utility
> >> >>> >> > can download file)
> >> >>> >> >
> >> >>> >> > Hi! I have a really baffling situation where my python tool
> cannot
> >> >>> >> > download a file and another tool (java based, use xrdcp) can
> >> download
> >> >>> >> > the same file ...
> >> >>> >> >
> >> >>> >> > the detailed logs for my cp are here :
> >> >>> >> > https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP
> >> >>> >> >
> >> >>> >> > the java based tool it seems that somehow ignores the external
> >> XRD_
> >> >>> >> > variables so i cannot get a log of cp process
> >> >>> >> >
> >> >>> >> > Could some expert take a look please and point me to a hint
> >> why my cp
> >> >>> >> > fails and the other tool can download just fine?
> >> >>> >> >
> >> >>> >> > Thank you!!
> >> >>> >> > Adrian
> >> >>> >> >
> >>
> >
> >
> > --
> > ----------------------------------------------
> > Adrian Sevcenco, Ph.D. |
> > Institute of Space Science - ISS, Romania |
> > adrian.sevcenco at {cern.ch,spacescience.ro} |
> > ----------------------------------------------
> >
> >
>
>
> --
> ----------------------------------------------
> Adrian Sevcenco, Ph.D. |
> Institute of Space Science - ISS, Romania |
> adrian.sevcenco at {cern.ch,spacescience.ro} |
> ----------------------------------------------
>
--
----------------------------------------------
Adrian Sevcenco, Ph.D. |
Institute of Space Science - ISS, Romania |
adrian.sevcenco at {cern.ch,spacescience.ro} |
----------------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
