On 2/4/20 4:23 PM, Michal Kamil Simon wrote:
> Hi Adiran,
Hi!
> I case of multi-source transfers the fallback is not implemented, please create a github issue for that.
done! https://github.com/xrootd/xrootd/issues/1128
thanks a lot!!
actually it was a little bit of surprise as it would seem that all
metalink clients use by default the multi source declaration for
fallback and secondary for parallel multi-source download
Personally (and i think also this apply to our experiment usage) the
multi-source parallel download is not important at all but the fallback
is paramount. I would say that the default behavior to be to parse all
sources until successful download and maybe a knob like --retries that
would specify how many times to parse the list of sources for a
successful download.
Moreover it seems that the hash is not verified otherwise there is no
explanation why it would have been accepted a file that have mangle
content and have a hash different from the metalink specified one ...
Is there a knob that i missed that should be enabled (in python
bindings) for the hash to be verified upon download?
If no, should i create another issue or add a comment to the above? (1128)
> Regarding the copying data out of ZIP archive, are using multi-source transfer to do that? If yes there
yeah .. i thought that it is required for fallback and while for normal
files everything went smooth i observer that those from zip files are
mangled .. initially i blamed a bad connection but then i received
reports from other users, so i managed to confirm that if i enable
multi-source then the output file is mangled (but have the correct size)
> is most likely a bug multi-source transfer logic that does not account for ZIP archives. Could you give
> me your xrdcp command you are executing and the source ZIP archive?
in the shared directory below
https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP
there is an meta4.tar.xz that contains the metalink used for downloading
(with/by python bindings)
as it has the full url with the token it should work directly
Thanks a lot again! :)
Adrian
>
> Cheers,
> Michal
> ________________________________________
> From: Adrian Sevcenco
> Sent: 03 February 2020 19:18
> To: [log in to unmask]; Michal Kamil Simon; Costin Grigoras; Nikola Hardi
> Subject: Re: XRootD py: there are zip files from each extraction does not take place
>
> On 2/3/20 5:48 PM, Adrian Sevcenco wrote:
>> Hi! It seems that there are some zip files from which the content is not
>> correctly extracted ... the files are the correct size but the content
>> is mangled ..
>> See the log at https://cernbox.cern.ch/index.php/s/JNaLKsaC5pyrhMP
>>
>> Let me know how can i assist with the debugging :)
>> (btw, this is the latest .2 tag)
> on more catch : the mangling happens when sourcelimit > 1
> we have multiple replicas for each file (and those appear in the
> metafile), and from my observations it would appear that if sourcelimit
> is not specified the fallback does not happen, so with sourcelimit =1 if
> the first replica is not available then the whole transfer is failed.
> Is this expected? my expectation is that fallback is present always and
> the sourcelimit > 1 only make a parallel chunked download between the
> specified source numbers available ...
>
> Thank you!
> Adrian
>
>
--
----------------------------------------------
Adrian Sevcenco, Ph.D. |
Institute of Space Science - ISS, Romania |
adrian.sevcenco at {cern.ch,spacescience.ro} |
----------------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
