Hi Oliver,
This is very interesting! Great to hear that someone has gone through some of the pain of figuring this out already :) Thanks for the link to the issue, I will try to pile on.
In any case, I would still be interested to know how feasible it would be to add XRootD protocol support to rclone.
Cheers,
Lincoln
________________________________
From: Oliver Freyermuth
Sent: Wednesday, August 10, 2022 12:47 PM
To: [log in to unmask]
Cc: Lincoln Bryant
Subject: Re: rclone support?
Hi Lincoln,
while it's not directly what you are asking for, I did some experiments with rclone and XRootD using XrdHttp, which may be of interest here.
Access works fine in general (mounting, copying etc.) using the generic "WebDAV" backend, but rclone breaks when a redirector is used:
https://github.com/rclone/rclone/issues/6029
One change was required in XrdHttp[0] such that the redirect HTTP code better matches the expectation of rclone and other tools (specs are nasty here), but rclone sadly still fails. However, the remaining issue (IMHO) is on the rclone end,
since it just gives up when redirected in many cases, completely ignoring the redirect. Sadly, the rclone devs have been silent on my question which kind of solution they would prefer.
If you are interested in this functionality, feel free to join in on the issue linked above.
However, the short news is: rclone works fine when used with an XRootD XrdHttp DTN directly via rclone's WebDAV backend, so if that is sufficient for your use case, it's something you can use "now" ;-).
Cheers and HTH,
Oliver
[0] https://github.com/xrootd/xrootd/issues/1638
PS: In case somebody wonders why rclone works with dCache WebDAV (even in distributed setups):
If I understand the code correctly, dCache proxies all traffic through the "redirector" in case such a client arrives, instead of redirecting to the corresponding transfer node,
likely to workaround such client bugs. Of course, this approach comes with some limitations on its own, so in my opinion, it would be preferable to fix such clients.
Am 10.08.22 um 19:16 schrieb Lincoln Bryant:
> Hi XRootD list,
>
> This is a little out there.. but I was wondering if it would be possible to add XRootD protocol support to Rclone? https://rclone.org/ <https://rclone.org/>
>
> The use case for me would be to sync a large volume of data from point A to point B, without a dedicated data management tool like Rucio.
>
> I see that there's an XRootD package as part of Go-Hep ( https://pkg.go.dev/go-hep.org/x/hep/xrootd <https://pkg.go.dev/go-hep.org/x/hep/xrootd> ), but I don't know how well maintained, supported, etc it is.
>
> Cheers,
> Lincoln
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1>
>
--
Oliver Freyermuth
Universität Bonn
Physikalisches Institut, Raum 1.047
Nußallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax: +49 228 73 7869
--
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
