 |
 |
Thank Oliver for explaining what this is all about. I was under the mistaken impression it was the other way around. Indeed, there is a go implementation of the xroot client, though I don't think it's a complete implementationbut likely sufficient for rclone use cases. I certainly am willing to advise the rclone team should they be interested in this. Andy On Wed, 10 Aug 2022, Oliver Freyermuth wrote: > Hi Andy, > > maybe my last mail was a bit misleading about what rclone really is (since I > did not provide any summary): > In fact, it's rather an open source client tool which tries to support many > storage backends, > especially also proprietary cloud things such as Dropbox, Google Drive, > OneDrive, Yandex etc. but also S3, WebDAV and various flavours of it > (ownCloud, nextCloud etc.). > > The special things it offers on top of upload / download is that it can also > mount (FUSE), copy between such configured "backends" (by streaming) and put > layers on top > (e.g. provide a local mount, and encrypt all data locally before uploading > encrypted chunks to things by public cloud providers). > To make things more confusing when trying to define what rclone is, it can > also serve local or remote data via some protocols (WebDAV etc.), i.e. > "re-export" data. > > So in a sense, it's like gfal with mounting capabilities, serving > capabilities, and support of much more protocols (and many messy protocols > requiring loads of quirks). > It's quite helpful to avoid proprietary closed-source clients in case you > can't avoid a commercial cloud provider for some reason. > > > In combination with XrdHttp, it proves quite useful to obtain a FUSE mount if > that is wanted. > > So what Lincoln was asking for was an implementation inside rclone to support > the XRootD protocol, such that it could be used as an XRootD client ? but I > think this this question is better asked to the rclone developers, > since the XRootD team provides full protocol documentation. > Since rclone is in Go, it would of course be easiest to have an existing > implementation of the XRootD protocol in Go, but as Lincoln already found > that seems to exist in go-hep/exp[0] developed by Sebastian Binet, > which can likely be leveraged by the rclone devs. So maybe opening an issue > in their GitHub project and linking that might be a good first step to gather > feedback from the rclone devs? > > > Thinking a bit more about the use case of syncing from A to B: I'm not sure > rclone is the correct tool for this. I did not find any information about it > being capable of triggering Third-Party-Copy transfers, > i.e. server-to-server copies (only for some special cases like "Google Drive > to Google Drive"), so unless I am mistaken, copies will always be streamed > through the client unless the "backend" implemented in rclone implements this > explicitly. > So in case you want good throughput, you may want to rather look at using > xrdcp, or gfal, or put an FTS / WebFTS in front. > > Cheers and HTH, > Oliver > > > [0] https://github.com/go-hep/exp > > Am 10.08.22 um 20:09 schrieb Andrew Hanushevsky: >> Hi Lincoln, >> >> OK, since I don't know much about rclone, it would appear that rclone has >> it's own request/response line protocol and there are such things as rclone >> servers. So, what Oliver is talking about is using a "backend" feature that >> uses webdav instad of the native rclone protocol. If so, the answer is yes >> it is possible in exactly the same way we added http/webdav protocol >> support to xrootd. The only requirement (and it's not that strict) is that >> rclone protocol have a differentiating "handshake" that would allow the >> xroot framework to direct connections too an rclone plugin. Then the plugin >> would handle the rclone protocol and do whatever it needs to do. >> >> Andy >> >> On Wed, 10 Aug 2022, Lincoln Bryant wrote: >> >>> Hi Oliver, >>> >>> This is very interesting! Great to hear that someone has gone through some >>> of the pain of figuring this out already :) Thanks for the link to the >>> issue, I will try to pile on. >>> >>> In any case, I would still be interested to know how feasible it would be >>> to add XRootD protocol support to rclone. >>> >>> Cheers, >>> Lincoln >>> >>> ________________________________ >>> From: Oliver Freyermuth >>> Sent: Wednesday, August 10, 2022 12:47 PM >>> To: [log in to unmask] >>> Cc: Lincoln Bryant >>> Subject: Re: rclone support? >>> >>> Hi Lincoln, >>> >>> while it's not directly what you are asking for, I did some experiments >>> with rclone and XRootD using XrdHttp, which may be of interest here. >>> >>> Access works fine in general (mounting, copying etc.) using the generic >>> "WebDAV" backend, but rclone breaks when a redirector is used: >>> https://github.com/rclone/rclone/issues/6029 >>> >>> One change was required in XrdHttp[0] such that the redirect HTTP code >>> better matches the expectation of rclone and other tools (specs are nasty >>> here), but rclone sadly still fails. However, the remaining issue (IMHO) >>> is on the rclone end, >>> since it just gives up when redirected in many cases, completely ignoring >>> the redirect. Sadly, the rclone devs have been silent on my question which >>> kind of solution they would prefer. >>> If you are interested in this functionality, feel free to join in on the >>> issue linked above. >>> >>> However, the short news is: rclone works fine when used with an XRootD >>> XrdHttp DTN directly via rclone's WebDAV backend, so if that is sufficient >>> for your use case, it's something you can use "now" ;-). >>> >>> Cheers and HTH, >>> Oliver >>> >>> [0] https://github.com/xrootd/xrootd/issues/1638 >>> >>> PS: In case somebody wonders why rclone works with dCache WebDAV (even in >>> distributed setups): >>> If I understand the code correctly, dCache proxies all traffic through >>> the "redirector" in case such a client arrives, instead of redirecting to >>> the corresponding transfer node, >>> likely to workaround such client bugs. Of course, this approach comes >>> with some limitations on its own, so in my opinion, it would be preferable >>> to fix such clients. >>> >>> Am 10.08.22 um 19:16 schrieb Lincoln Bryant: >>>> Hi XRootD list, >>>> >>>> This is a little out there.. but I was wondering if it would be possible >>>> to add XRootD protocol support to Rclone? https://rclone.org/ >>>> <https://rclone.org/> >>>> >>>> The use case for me would be to sync a large volume of data from point A >>>> to point B, without a dedicated data management tool like Rucio. >>>> >>>> I see that there's an XRootD package as part of Go-Hep ( >>>> https://pkg.go.dev/go-hep.org/x/hep/xrootd >>>> <https://pkg.go.dev/go-hep.org/x/hep/xrootd> ), but I don't know how well >>>> maintained, supported, etc it is. >>>> >>>> Cheers, >>>> Lincoln >>>> >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>> -- >> --- >>>> >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>>> <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1> >>>> >>> >>> >>> -- >>> Oliver Freyermuth >>> Universität Bonn >>> Physikalisches Institut, Raum 1.047 >>> Nußallee 12 >>> 53115 Bonn >>> -- >>> Tel.: +49 228 73 2367 >>> Fax: +49 228 73 7869 >>> -- >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> > > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.047 > Nußallee 12 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1