 |
 |
On 4/1/19 6:07 PM, Adrian Sevcenco wrote: > On 4/1/19 5:17 PM, Michal Kamil Simon wrote: >> Hi Adrian, > Hi! > >> Regarding the parsing problem of your URLs, it's not a >> problem of our implementation. As a metalink is basically >> an xml file you simply need to escape certain characters >> or you need to use CDATA. >> >> Try something like: >> >> <url><*![CDATA[* /your URL with alice token/ *]]>*</url> > well, i tried :( see > https://sev.spacescience.ro/adrian/meta/alice_cern.ch_user_a_asevcenc_testfile.meta4 > > but i get : > xrdcp -N -p -P -f -v alice_cern.ch_user_a_asevcenc_testfile.meta4 testfile_downloaded.txt > Prepare: [ERROR] Received corrupted data: Malformed or corrupted metalink file. > >> in your metalink file (it did work for me ;-) > so, what am i missing? there where no spaces after CDATA[ and before ]] so, i can confirm that xrdcp works with meta4 file but i have problems with client.CopyProcess() i do : process.add_job(url_src["url"], url_dst["url"], force = False, posc = True, mkdir = True, chunksize = 4194304, parallelchunks = 1, sourcelimit = 1) where : src,dst is alice_cern.ch_user_a_asevcenc_testfile.meta4 /home.hdd/adrian/work-GRID/jalien_cli_py/testfile_downloaded.txt but i get nothing .. not even the print(source,target) from CopyProgressHandler.begin Any idea where can i look? Thanks a lot! Adrian > > Thanks a lot! > Adrian > > >> >> Cheers, >> Michal >> ________________________________________ >> From: Adrian Sevcenco >> Sent: 01 April 2019 15:30 >> To: Michal Kamil Simon; [log in to unmask] >> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed >> >> On 4/1/19 4:12 PM, Michal Kamil Simon wrote: >> > Hi Adrian, >> > >> > You cannot use a metalink as a destination, but you can use >> yeah, i know that :P >> >> > a metalink as a source for a remote destination ;-) >> this the concept that i did not thought about :) >> >> Thanks! >> Adrian >> >> >> > >> > Cheers, >> > Michal >> > ________________________________________ >> > From: Adrian Sevcenco >> > Sent: 01 April 2019 15:02 >> > To: Michal Kamil Simon; [log in to unmask] >> > Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed >> > >> > On 4/1/19 3:20 PM, Michal Kamil Simon wrote: >> >> Hi Adrian, >> >> >> >> The name attribute specifies the destination file name >> >> (or a relative path to the file as in the example), however keep >> >> in mind that the destination might be remote. >> > hmmm, i thought that metalink files are only for download >> > >> >> To give you an example: >> >> >> >> / xrdcp input.meta4 root://hostname//path/dir/ >> >> >> >> and say the name attribute is /subdir/file.txt/ >> >> >> >> the effective destination will be: >> >> /root://hostname//path/dir/subdir/file.txt/ >> >> >> >> however if you overwrite the destination file name in /xrdcp/ command >> >> line, e.g.: >> >> >> >> xrdcp input.meta4 /root://hostname//path/dir/file.txt/ >> >> >> >> the name attribute will be ignored. >> >> >> >> Hope this makes it more clear :-) >> > yeah, i just ignored the fact that it can be used with remote destinations >> > >> > well, thanks for info :) >> > looking forward for the fix :) >> > >> > Thanks a lot! >> > Adrian >> > >> > >> >> >> >> Cheers, >> >> Michal >> >> ________________________________________ >> >> From: Adrian Sevcenco >> >> Sent: 01 April 2019 13:22 >> >> To: Michal Kamil Simon; [log in to unmask] >> >> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed >> >> >> >> On 4/1/19 1:10 PM, Michal Kamil Simon wrote: >> >> > Hi Adrian, >> >> Hi! >> >> >> >> > Thanks for providing an example metalink, indeed >> >> > it seems there is a problem with parsing a metalink >> >> > that contains urls with alice tokens. I'll provide a fix for >> >> > that. >> >> Thanks a lot!! >> >> >> >> > Regarding the destination, you are suppose to specify >> >> > possible sources in the metalink, but it must not contain >> >> > the destination, which has to be specified as for any other >> >> > copy job. >> >> well, https://tools.ietf.org/html/rfc5854#section-4.1.2.1 >> >> metalink:file elements MUST have a "name" attribute, which contains >> >> the local file name to which the downloaded file will be written. >> >> >> >> This is why i said that destination should not be required... >> >> but of course is not hard for me to take care to use the same destination >> >> file as it is written in the meta4 file >> >> >> >> Thanks a lot! >> >> Adrian >> >> >> >> >> >> > >> >> > Cheers, >> >> > Michal >> >> > ________________________________________ >> >> > From: Adrian Sevcenco >> >> > Sent: 31 March 2019 00:09 >> >> > To: Michal Kamil Simon; [log in to unmask] >> >> > Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed >> >> > >> >> > On 3/28/19 5:31 PM, Michal Kamil Simon wrote: >> >> >> Hi Adrian, >> >> > Hi Michal! >> >> > >> >> >> If (for now) you simply want to download a file having different >> >> >> replicas (with >> >> >> different names, etc.) a metalink is what you need. Have a look at: >> >> > cool! it is perfect for my need >> >> > >> >> >> https://en.wikipedia.org/wiki/Metalink#Example_Metalink_4.0_.meta4_file >> >> >> >> >> >> you can create a metalink containing different URLs and assign priorities >> >> >> to them. You could e.g. generate the metalink from python and then use a >> >> >> local metalink as source in the CopyProcess. >> >> >> >> >> >> Once we release 4.9.1 you will be also able to do a multi-source download >> >> >> using a metalink. >> >> >> >> >> >> However, this wont work for uploads. >> >> > of course! >> >> > >> >> > I made one for testing, >> >> > https://sev.spacescience.ro/adrian/meta/alice_cern.ch_user_a_asevcenc_testfile.meta4 >> >> > >> >> > but it seems that there are problems : >> >> > 1. the url part is not parsed properly >> >> > >> >> > 2. the usage require a destination (but that is required and included in >> >> > the metalink) - so maybe if the source is meta4 the destination is taken >> >> > from the meta file and no longer required in cli? >> >> > >> >> > I am sure that there is a problem with the url parsing because the same >> >> > url works for a normal add_job copy request >> >> > >> >> > Additional question for when it will be working : >> >> > for the python part - add_Job, will take as source this .meta4 file? >> >> > >> >> > Thank you!! >> >> > Adrian >> >> > >> >> >> >> >> >> Cheers, >> >> >> Michal >> >> >> ________________________________________ >> >> >> From: Adrian Sevcenco >> >> >> Sent: 28 March 2019 16:03 >> >> >> To: Michal Kamil Simon; [log in to unmask] >> >> >> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed >> >> >> >> >> >> On 3/28/19 4:35 PM, Michal Kamil Simon wrote: >> >> >> > Hi Adrian, >> >> >> Hi! >> >> >> >> >> >> > My impression (correct me if I'm wrong) is that you are trying to do >> >> >> a multi-source transfer, >> >> >> > right? >> >> >> well, that would have been the optimal result, but at this moment i was >> >> >> just trying >> >> >> to download a file while trying all replicas >> >> >> >> >> >> > If so, you need to specify in the CopyProcess the sourceLimit, and as >> >> >> a source you need >> >> >> > to specify a metalink containing all the replicas, or a manager (in >> >> >> this case the client will >> >> >> > figure out what the replicas are by itself by using locate). >> >> >> my case is a little special ... the same file have multiple replicas >> >> >> that have _different_ physical names >> >> >> and authz envelopes ... >> >> >> as a client i have all required info.. i know for my logical file name >> >> >> /alice/cern.ch/user/a/asevcenc/testfile >> >> >> that the actual links are : >> >> >> root://eosalice.cern.ch:1094//06/17419/a7f1c690-4e49-11e9-8a64-0242266c4d21?_LONG_STRING_OF_OPTIONS_ >> >> >> and >> >> >> root://ipngridxrd0.in2p3.fr:1094//06/17419/a7f1c690-4e49-11e9-8a64-0242266c4d21?_LONG_STRING_OF_OPTIONS_ >> >> >> >> >> >> how can i create/what is the format of a metalink? >> >> >> can i somehow put toghether the remote links from above? >> >> >> >> >> >> would the same mechanism work for upload? >> >> >> >> >> >> > I just pushed a patch for the client that is enabling multi-source >> >> >> download in python bindings >> >> >> > (63e9604) and will port it for 4.9.1. >> >> >> great! thanks a lot!!! >> >> >> >> >> >> Adrian >> >> >> >> >> >> >> >> >> > >> >> >> > Cheers, >> >> >> > Michal >> >> >> > ________________________________________ >> >> >> > From: [log in to unmask] [[log in to unmask]] on >> >> >> behalf of Adrian Sevcenco [[log in to unmask]] >> >> >> > Sent: 28 March 2019 14:22 >> >> >> > To: [log in to unmask] >> >> >> > Subject: xrootd python :: CopyProcess - stop the queue if a job succeed >> >> >> > >> >> >> > Hi! In my use case (ALICE), to download a file, i get all replicas >> >> >> > and i add them as copy jobs to a CopyProcess .. >> >> >> > I would like to stop the CopyProcess when a job is succesful .. >> >> >> > Is there a way to do this? >> >> >> > Maybe in MyCopyProgressHandler.end() to check if results['status'] have >> >> >> > a "status: 0" and cancel all CopyProcess? >> >> >> > Is there a way to stop a CopyProcess? or other way to take into >> >> >> account the posibility >> >> >> > of multiple replicas? >> >> >> > >> >> >> > Thank you! >> >> >> > Adrian >> >> >> > >> >> >> > >> >> >> > ######################################################################## >> >> >> > Use REPLY-ALL to reply to list >> >> >> > >> >> >> > To unsubscribe from the XROOTD-L list, click the following link: >> >> >> > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> >> >> > >> >> > >> >> > >> >> >> >> >> > >> > >> > >> >> > > > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1