 |
 |
Hi Andy, maybe my last mail was a bit misleading about what rclone really is (since I did not provide any summary): In fact, it's rather an open source client tool which tries to support many storage backends, especially also proprietary cloud things such as Dropbox, Google Drive, OneDrive, Yandex etc. but also S3, WebDAV and various flavours of it (ownCloud, nextCloud etc.). The special things it offers on top of upload / download is that it can also mount (FUSE), copy between such configured "backends" (by streaming) and put layers on top (e.g. provide a local mount, and encrypt all data locally before uploading encrypted chunks to things by public cloud providers). To make things more confusing when trying to define what rclone is, it can also serve local or remote data via some protocols (WebDAV etc.), i.e. "re-export" data. So in a sense, it's like gfal with mounting capabilities, serving capabilities, and support of much more protocols (and many messy protocols requiring loads of quirks). It's quite helpful to avoid proprietary closed-source clients in case you can't avoid a commercial cloud provider for some reason. In combination with XrdHttp, it proves quite useful to obtain a FUSE mount if that is wanted. So what Lincoln was asking for was an implementation inside rclone to support the XRootD protocol, such that it could be used as an XRootD client — but I think this this question is better asked to the rclone developers, since the XRootD team provides full protocol documentation. Since rclone is in Go, it would of course be easiest to have an existing implementation of the XRootD protocol in Go, but as Lincoln already found that seems to exist in go-hep/exp[0] developed by Sebastian Binet, which can likely be leveraged by the rclone devs. So maybe opening an issue in their GitHub project and linking that might be a good first step to gather feedback from the rclone devs? Thinking a bit more about the use case of syncing from A to B: I'm not sure rclone is the correct tool for this. I did not find any information about it being capable of triggering Third-Party-Copy transfers, i.e. server-to-server copies (only for some special cases like "Google Drive to Google Drive"), so unless I am mistaken, copies will always be streamed through the client unless the "backend" implemented in rclone implements this explicitly. So in case you want good throughput, you may want to rather look at using xrdcp, or gfal, or put an FTS / WebFTS in front. Cheers and HTH, Oliver [0] https://github.com/go-hep/exp Am 10.08.22 um 20:09 schrieb Andrew Hanushevsky: > Hi Lincoln, > > OK, since I don't know much about rclone, it would appear that rclone has it's own request/response line protocol and there are such things as rclone servers. So, what Oliver is talking about is using a "backend" feature that uses webdav instad of the native rclone protocol. If so, the answer is yes it is possible in exactly the same way we added http/webdav protocol support to xrootd. The only requirement (and it's not that strict) is that rclone protocol have a differentiating "handshake" that would allow the xroot framework to direct connections too an rclone plugin. Then the plugin would handle the rclone protocol and do whatever it needs to do. > > Andy > > On Wed, 10 Aug 2022, Lincoln Bryant wrote: > >> Hi Oliver, >> >> This is very interesting! Great to hear that someone has gone through some of the pain of figuring this out already :) Thanks for the link to the issue, I will try to pile on. >> >> In any case, I would still be interested to know how feasible it would be to add XRootD protocol support to rclone. >> >> Cheers, >> Lincoln >> >> ________________________________ >> From: Oliver Freyermuth >> Sent: Wednesday, August 10, 2022 12:47 PM >> To: [log in to unmask] >> Cc: Lincoln Bryant >> Subject: Re: rclone support? >> >> Hi Lincoln, >> >> while it's not directly what you are asking for, I did some experiments with rclone and XRootD using XrdHttp, which may be of interest here. >> >> Access works fine in general (mounting, copying etc.) using the generic "WebDAV" backend, but rclone breaks when a redirector is used: >> https://github.com/rclone/rclone/issues/6029 >> >> One change was required in XrdHttp[0] such that the redirect HTTP code better matches the expectation of rclone and other tools (specs are nasty here), but rclone sadly still fails. However, the remaining issue (IMHO) is on the rclone end, >> since it just gives up when redirected in many cases, completely ignoring the redirect. Sadly, the rclone devs have been silent on my question which kind of solution they would prefer. >> If you are interested in this functionality, feel free to join in on the issue linked above. >> >> However, the short news is: rclone works fine when used with an XRootD XrdHttp DTN directly via rclone's WebDAV backend, so if that is sufficient for your use case, it's something you can use "now" ;-). >> >> Cheers and HTH, >> Oliver >> >> [0] https://github.com/xrootd/xrootd/issues/1638 >> >> PS: In case somebody wonders why rclone works with dCache WebDAV (even in distributed setups): >> If I understand the code correctly, dCache proxies all traffic through the "redirector" in case such a client arrives, instead of redirecting to the corresponding transfer node, >> likely to workaround such client bugs. Of course, this approach comes with some limitations on its own, so in my opinion, it would be preferable to fix such clients. >> >> Am 10.08.22 um 19:16 schrieb Lincoln Bryant: >>> Hi XRootD list, >>> >>> This is a little out there.. but I was wondering if it would be possible to add XRootD protocol support to Rclone? https://rclone.org/ <https://rclone.org/> >>> >>> The use case for me would be to sync a large volume of data from point A to point B, without a dedicated data management tool like Rucio. >>> >>> I see that there's an XRootD package as part of Go-Hep ( https://pkg.go.dev/go-hep.org/x/hep/xrootd <https://pkg.go.dev/go-hep.org/x/hep/xrootd> ), but I don't know how well maintained, supported, etc it is. >>> >>> Cheers, >>> Lincoln >>> >>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > --- >>> >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1> >>> >> >> >> -- >> Oliver Freyermuth >> Universität Bonn >> Physikalisches Institut, Raum 1.047 >> Nußallee 12 >> 53115 Bonn >> -- >> Tel.: +49 228 73 2367 >> Fax: +49 228 73 7869 >> -- >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 -- ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1