XROOTD-L Archives

Support use of xrootd by HEP experiments

XROOTD-L@LISTSERV.SLAC.STANFORD.EDU

Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Adrian Sevcenco <[log in to unmask]>
Reply To:
Support use of xrootd by HEP experiments <[log in to unmask]>
Date:
Mon, 1 Apr 2019 16:30:00 +0300
Content-Type:
multipart/signed
Parts/Attachments:
text/plain (7 kB) , smime.p7s (7 kB)
On 4/1/19 4:12 PM, Michal Kamil Simon wrote:
> Hi Adrian,
> 
> You cannot use a metalink as a destination, but you can use
yeah, i know that :P

> a metalink as a source for a remote destination ;-)
this the concept that i did not thought about :)

Thanks!
Adrian


> 
> Cheers,
> Michal
> ________________________________________
> From: Adrian Sevcenco
> Sent: 01 April 2019 15:02
> To: Michal Kamil Simon; [log in to unmask]
> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed
> 
> On 4/1/19 3:20 PM, Michal Kamil Simon wrote:
>> Hi Adrian,
>>
>> The name attribute specifies the destination file name
>> (or a relative path to the file as in the example), however keep
>> in mind that the destination might be remote.
> hmmm, i thought that metalink files are only for download
> 
>> To give you an example:
>>
>> /   xrdcp input.meta4   root://hostname//path/dir/
>>
>>      and say the name attribute is /subdir/file.txt/
>>
>> the effective destination will be:
>> /root://hostname//path/dir/subdir/file.txt/
>>
>> however if you overwrite the destination file name in /xrdcp/ command
>> line, e.g.:
>>
>>       xrdcp input.meta4 /root://hostname//path/dir/file.txt/
>>
>> the name attribute will be ignored.
>>
>> Hope this makes it more clear :-)
> yeah, i just ignored the fact that it can be used with remote destinations
> 
> well, thanks for info :)
> looking forward for the fix :)
> 
> Thanks a lot!
> Adrian
> 
> 
>>
>> Cheers,
>> Michal
>> ________________________________________
>> From: Adrian Sevcenco
>> Sent: 01 April 2019 13:22
>> To: Michal Kamil Simon; [log in to unmask]
>> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed
>>
>> On 4/1/19 1:10 PM, Michal Kamil Simon wrote:
>>   > Hi Adrian,
>> Hi!
>>
>>   > Thanks for providing an example metalink, indeed
>>   > it seems there is a problem with parsing a metalink
>>   > that contains urls with alice tokens. I'll provide a fix for
>>   > that.
>> Thanks a lot!!
>>
>>   > Regarding the destination, you are suppose to specify
>>   > possible sources in the metalink, but it must not contain
>>   > the destination, which has to be specified as for any other
>>   > copy job.
>> well, https://tools.ietf.org/html/rfc5854#section-4.1.2.1
>> metalink:file elements MUST have a "name" attribute, which contains
>> the local file name to which the downloaded file will be written.
>>
>> This is why i said that destination should not be required...
>> but of course is not hard for me to take care to use the same destination
>> file as it is written in the meta4 file
>>
>> Thanks a lot!
>> Adrian
>>
>>
>>   >
>>   > Cheers,
>>   > Michal
>>   > ________________________________________
>>   > From: Adrian Sevcenco
>>   > Sent: 31 March 2019 00:09
>>   > To: Michal Kamil Simon; [log in to unmask]
>>   > Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed
>>   >
>>   > On 3/28/19 5:31 PM, Michal Kamil Simon wrote:
>>   >> Hi Adrian,
>>   > Hi Michal!
>>   >
>>   >> If (for now) you simply want to download a file having different
>>   >> replicas (with
>>   >> different names, etc.) a metalink is what you need. Have a look at:
>>   > cool! it is perfect for my need
>>   >
>>   >> https://en.wikipedia.org/wiki/Metalink#Example_Metalink_4.0_.meta4_file
>>   >>
>>   >> you can create a metalink containing different URLs and assign priorities
>>   >> to them. You could e.g. generate the metalink from python and then use a
>>   >> local metalink as source in the CopyProcess.
>>   >>
>>   >> Once we release 4.9.1 you will be also able to do a multi-source download
>>   >> using a metalink.
>>   >>
>>   >> However, this wont work for uploads.
>>   > of course!
>>   >
>>   > I made one for testing,
>>   > https://sev.spacescience.ro/adrian/meta/alice_cern.ch_user_a_asevcenc_testfile.meta4
>>   >
>>   > but it seems that there are problems :
>>   > 1. the url part is not parsed properly
>>   >
>>   > 2. the usage require a destination (but that is required and included in
>>   > the metalink) - so maybe if the source is meta4 the destination is taken
>>   > from the meta file and no longer required in cli?
>>   >
>>   > I am sure that there is a problem with the url parsing because the same
>>   > url works for a normal add_job copy request
>>   >
>>   > Additional question for when it will be working :
>>   > for the python part - add_Job, will take as source this .meta4 file?
>>   >
>>   > Thank you!!
>>   > Adrian
>>   >
>>   >>
>>   >> Cheers,
>>   >> Michal
>>   >> ________________________________________
>>   >> From: Adrian Sevcenco
>>   >> Sent: 28 March 2019 16:03
>>   >> To: Michal Kamil Simon; [log in to unmask]
>>   >> Subject: Re: xrootd python :: CopyProcess - stop the queue if a job succeed
>>   >>
>>   >> On 3/28/19 4:35 PM, Michal Kamil Simon wrote:
>>   >> > Hi Adrian,
>>   >> Hi!
>>   >>
>>   >> > My impression (correct me if I'm wrong) is that you are trying to do
>>   >> a multi-source transfer,
>>   >> > right?
>>   >> well, that would have been the optimal result, but at this moment i was
>>   >> just trying
>>   >> to download a file while trying all replicas
>>   >>
>>   >> > If so, you need to specify in the CopyProcess the sourceLimit, and as
>>   >> a source you need
>>   >> > to specify a metalink containing all the replicas, or a manager (in
>>   >> this case the client will
>>   >> > figure out what the replicas are by itself by using locate).
>>   >> my case is a little special ... the same file have multiple replicas
>>   >> that have _different_ physical names
>>   >> and authz envelopes ...
>>   >> as a client i have all required info.. i know for my logical file name
>>   >> /alice/cern.ch/user/a/asevcenc/testfile
>>   >> that the actual links are :
>>   >> root://eosalice.cern.ch:1094//06/17419/a7f1c690-4e49-11e9-8a64-0242266c4d21?_LONG_STRING_OF_OPTIONS_
>>   >> and
>>   >> root://ipngridxrd0.in2p3.fr:1094//06/17419/a7f1c690-4e49-11e9-8a64-0242266c4d21?_LONG_STRING_OF_OPTIONS_
>>   >>
>>   >> how can i create/what is the format of a metalink?
>>   >> can i somehow put toghether the remote links from above?
>>   >>
>>   >> would the same mechanism work for upload?
>>   >>
>>   >> > I just pushed a patch for the client that is enabling multi-source
>>   >> download in python bindings
>>   >> > (63e9604) and will port it for 4.9.1.
>>   >> great! thanks a lot!!!
>>   >>
>>   >> Adrian
>>   >>
>>   >>
>>   >> >
>>   >> > Cheers,
>>   >> > Michal
>>   >> > ________________________________________
>>   >> > From: [log in to unmask] [[log in to unmask]] on
>>   >> behalf of Adrian Sevcenco [[log in to unmask]]
>>   >> > Sent: 28 March 2019 14:22
>>   >> > To: [log in to unmask]
>>   >> > Subject: xrootd python :: CopyProcess - stop the queue if a job succeed
>>   >> >
>>   >> > Hi! In my use case (ALICE), to download a file, i get all replicas
>>   >> > and i add them as copy jobs to a CopyProcess ..
>>   >> > I would like to stop the CopyProcess when a job is succesful ..
>>   >> > Is there a way to do this?
>>   >> > Maybe in MyCopyProgressHandler.end() to check if results['status'] have
>>   >> > a "status: 0" and cancel all CopyProcess?
>>   >> > Is there a way to stop a CopyProcess? or other way to take into
>>   >> account the posibility
>>   >> > of multiple replicas?
>>   >> >
>>   >> > Thank you!
>>   >> > Adrian
>>   >> >
>>   >> >
>>   >> > ########################################################################
>>   >> > Use REPLY-ALL to reply to list
>>   >> >
>>   >> > To unsubscribe from the XROOTD-L list, click the following link:
>>   >> > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>   >> >
>>   >
>>   >
>>
>>
> 
> 
> 



########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1


ATOM RSS1 RSS2