 |
 |
Hi Lincoln, OK, since I don't know much about rclone, it would appear that rclone has it's own request/response line protocol and there are such things as rclone servers. So, what Oliver is talking about is using a "backend" feature that uses webdav instad of the native rclone protocol. If so, the answer is yes it is possible in exactly the same way we added http/webdav protocol support to xrootd. The only requirement (and it's not that strict) is that rclone protocol have a differentiating "handshake" that would allow the xroot framework to direct connections too an rclone plugin. Then the plugin would handle the rclone protocol and do whatever it needs to do. Andy On Wed, 10 Aug 2022, Lincoln Bryant wrote: > Hi Oliver, > > This is very interesting! Great to hear that someone has gone through some of the pain of figuring this out already :) Thanks for the link to the issue, I will try to pile on. > > In any case, I would still be interested to know how feasible it would be to add XRootD protocol support to rclone. > > Cheers, > Lincoln > > ________________________________ > From: Oliver Freyermuth > Sent: Wednesday, August 10, 2022 12:47 PM > To: [log in to unmask] > Cc: Lincoln Bryant > Subject: Re: rclone support? > > Hi Lincoln, > > while it's not directly what you are asking for, I did some experiments with rclone and XRootD using XrdHttp, which may be of interest here. > > Access works fine in general (mounting, copying etc.) using the generic "WebDAV" backend, but rclone breaks when a redirector is used: > https://github.com/rclone/rclone/issues/6029 > > One change was required in XrdHttp[0] such that the redirect HTTP code better matches the expectation of rclone and other tools (specs are nasty here), but rclone sadly still fails. However, the remaining issue (IMHO) is on the rclone end, > since it just gives up when redirected in many cases, completely ignoring the redirect. Sadly, the rclone devs have been silent on my question which kind of solution they would prefer. > If you are interested in this functionality, feel free to join in on the issue linked above. > > However, the short news is: rclone works fine when used with an XRootD XrdHttp DTN directly via rclone's WebDAV backend, so if that is sufficient for your use case, it's something you can use "now" ;-). > > Cheers and HTH, > Oliver > > [0] https://github.com/xrootd/xrootd/issues/1638 > > PS: In case somebody wonders why rclone works with dCache WebDAV (even in distributed setups): > If I understand the code correctly, dCache proxies all traffic through the "redirector" in case such a client arrives, instead of redirecting to the corresponding transfer node, > likely to workaround such client bugs. Of course, this approach comes with some limitations on its own, so in my opinion, it would be preferable to fix such clients. > > Am 10.08.22 um 19:16 schrieb Lincoln Bryant: >> Hi XRootD list, >> >> This is a little out there.. but I was wondering if it would be possible to add XRootD protocol support to Rclone? https://rclone.org/ <https://rclone.org/> >> >> The use case for me would be to sync a large volume of data from point A to point B, without a dedicated data management tool like Rucio. >> >> I see that there's an XRootD package as part of Go-Hep ( https://pkg.go.dev/go-hep.org/x/hep/xrootd <https://pkg.go.dev/go-hep.org/x/hep/xrootd> ), but I don't know how well maintained, supported, etc it is. >> >> Cheers, >> Lincoln >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --- >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1> >> > > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.047 > Nußallee 12 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1