Thank Oliver for explaining what this is all about. I was under the
mistaken impression it was the other way around. Indeed, there is a go
implementation of the xroot client, though I don't think it's a complete
implementationbut likely sufficient for rclone use cases. I certainly am
willing to advise the rclone team should they be interested in this.
Andy
On Wed, 10 Aug 2022, Oliver Freyermuth wrote:
> Hi Andy,
>
> maybe my last mail was a bit misleading about what rclone really is (since I
> did not provide any summary):
> In fact, it's rather an open source client tool which tries to support many
> storage backends,
> especially also proprietary cloud things such as Dropbox, Google Drive,
> OneDrive, Yandex etc. but also S3, WebDAV and various flavours of it
> (ownCloud, nextCloud etc.).
>
> The special things it offers on top of upload / download is that it can also
> mount (FUSE), copy between such configured "backends" (by streaming) and put
> layers on top
> (e.g. provide a local mount, and encrypt all data locally before uploading
> encrypted chunks to things by public cloud providers).
> To make things more confusing when trying to define what rclone is, it can
> also serve local or remote data via some protocols (WebDAV etc.), i.e.
> "re-export" data.
>
> So in a sense, it's like gfal with mounting capabilities, serving
> capabilities, and support of much more protocols (and many messy protocols
> requiring loads of quirks).
> It's quite helpful to avoid proprietary closed-source clients in case you
> can't avoid a commercial cloud provider for some reason.
>
>
> In combination with XrdHttp, it proves quite useful to obtain a FUSE mount if
> that is wanted.
>
> So what Lincoln was asking for was an implementation inside rclone to support
> the XRootD protocol, such that it could be used as an XRootD client ? but I
> think this this question is better asked to the rclone developers,
> since the XRootD team provides full protocol documentation.
> Since rclone is in Go, it would of course be easiest to have an existing
> implementation of the XRootD protocol in Go, but as Lincoln already found
> that seems to exist in go-hep/exp[0] developed by Sebastian Binet,
> which can likely be leveraged by the rclone devs. So maybe opening an issue
> in their GitHub project and linking that might be a good first step to gather
> feedback from the rclone devs?
>
>
> Thinking a bit more about the use case of syncing from A to B: I'm not sure
> rclone is the correct tool for this. I did not find any information about it
> being capable of triggering Third-Party-Copy transfers,
> i.e. server-to-server copies (only for some special cases like "Google Drive
> to Google Drive"), so unless I am mistaken, copies will always be streamed
> through the client unless the "backend" implemented in rclone implements this
> explicitly.
> So in case you want good throughput, you may want to rather look at using
> xrdcp, or gfal, or put an FTS / WebFTS in front.
>
> Cheers and HTH,
> Oliver
>
>
> [0] https://github.com/go-hep/exp
>
> Am 10.08.22 um 20:09 schrieb Andrew Hanushevsky:
>> Hi Lincoln,
>>
>> OK, since I don't know much about rclone, it would appear that rclone has
>> it's own request/response line protocol and there are such things as rclone
>> servers. So, what Oliver is talking about is using a "backend" feature that
>> uses webdav instad of the native rclone protocol. If so, the answer is yes
>> it is possible in exactly the same way we added http/webdav protocol
>> support to xrootd. The only requirement (and it's not that strict) is that
>> rclone protocol have a differentiating "handshake" that would allow the
>> xroot framework to direct connections too an rclone plugin. Then the plugin
>> would handle the rclone protocol and do whatever it needs to do.
>>
>> Andy
>>
>> On Wed, 10 Aug 2022, Lincoln Bryant wrote:
>>
>>> Hi Oliver,
>>>
>>> This is very interesting! Great to hear that someone has gone through some
>>> of the pain of figuring this out already :) Thanks for the link to the
>>> issue, I will try to pile on.
>>>
>>> In any case, I would still be interested to know how feasible it would be
>>> to add XRootD protocol support to rclone.
>>>
>>> Cheers,
>>> Lincoln
>>>
>>> ________________________________
>>> From: Oliver Freyermuth
>>> Sent: Wednesday, August 10, 2022 12:47 PM
>>> To: [log in to unmask]
>>> Cc: Lincoln Bryant
>>> Subject: Re: rclone support?
>>>
>>> Hi Lincoln,
>>>
>>> while it's not directly what you are asking for, I did some experiments
>>> with rclone and XRootD using XrdHttp, which may be of interest here.
>>>
>>> Access works fine in general (mounting, copying etc.) using the generic
>>> "WebDAV" backend, but rclone breaks when a redirector is used:
>>> https://github.com/rclone/rclone/issues/6029
>>>
>>> One change was required in XrdHttp[0] such that the redirect HTTP code
>>> better matches the expectation of rclone and other tools (specs are nasty
>>> here), but rclone sadly still fails. However, the remaining issue (IMHO)
>>> is on the rclone end,
>>> since it just gives up when redirected in many cases, completely ignoring
>>> the redirect. Sadly, the rclone devs have been silent on my question which
>>> kind of solution they would prefer.
>>> If you are interested in this functionality, feel free to join in on the
>>> issue linked above.
>>>
>>> However, the short news is: rclone works fine when used with an XRootD
>>> XrdHttp DTN directly via rclone's WebDAV backend, so if that is sufficient
>>> for your use case, it's something you can use "now" ;-).
>>>
>>> Cheers and HTH,
>>> Oliver
>>>
>>> [0] https://github.com/xrootd/xrootd/issues/1638
>>>
>>> PS: In case somebody wonders why rclone works with dCache WebDAV (even in
>>> distributed setups):
>>> If I understand the code correctly, dCache proxies all traffic through
>>> the "redirector" in case such a client arrives, instead of redirecting to
>>> the corresponding transfer node,
>>> likely to workaround such client bugs. Of course, this approach comes
>>> with some limitations on its own, so in my opinion, it would be preferable
>>> to fix such clients.
>>>
>>> Am 10.08.22 um 19:16 schrieb Lincoln Bryant:
>>>> Hi XRootD list,
>>>>
>>>> This is a little out there.. but I was wondering if it would be possible
>>>> to add XRootD protocol support to Rclone? https://rclone.org/
>>>> <https://rclone.org/>
>>>>
>>>> The use case for me would be to sync a large volume of data from point A
>>>> to point B, without a dedicated data management tool like Rucio.
>>>>
>>>> I see that there's an XRootD package as part of Go-Hep (
>>>> https://pkg.go.dev/go-hep.org/x/hep/xrootd
>>>> <https://pkg.go.dev/go-hep.org/x/hep/xrootd> ), but I don't know how well
>>>> maintained, supported, etc it is.
>>>>
>>>> Cheers,
>>>> Lincoln
>>>>
>>>>
>>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> --
>> ---
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>> <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1>
>>>>
>>>
>>>
>>> --
>>> Oliver Freyermuth
>>> Universität Bonn
>>> Physikalisches Institut, Raum 1.047
>>> Nußallee 12
>>> 53115 Bonn
>>> --
>>> Tel.: +49 228 73 2367
>>> Fax: +49 228 73 7869
>>> --
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>
>
> --
> Oliver Freyermuth
> Universität Bonn
> Physikalisches Institut, Raum 1.047
> Nußallee 12
> 53115 Bonn
> --
> Tel.: +49 228 73 2367
> Fax: +49 228 73 7869
> --
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|