 |
 |
Hi Wei, yes, I see, and I have a good understanding of S3, coming from the evolution of Davix. S3 is built on HTTP, yes, however it adds many bits and pieces, as you discovered. IMO the best solution to this is to make an XrdHttp plugin in order to override its behaviour towards being S3-ish. Making the plugin is pretty easy, the interface is rather simple, what would become heavier IMO is following up with the feature requests that inevitably will come when advertising S3 pragmatic compatibility. Fabrizio Il 11/04/2022 10:52, Yang, Wei ha scritto: > Hi Fabrizio, > > I am trying to understand how far away is XrdHTTP from a S3 object store. I know XrdHttp will work with 'requests', etc. but by far the most popular object store is s3. Rubin is one of the experiments that widely uses s3 (they claim they support webdav as well). They don't use all the features of s3 but a subset of basic functions. Things like ACL and multipart upload are mentioned but not in high priority. > > I am not sure I fully understand the difference between HTTP(s) and S3. I thought that S3 is built on top of HTTPs. > > So I am trying boto3 against XrdHTTP. It both surprises and not surprise to me that some of the things are actually working. e.g.download_file(), upload_file() (if I remove the emoji from the XrdHTTP response), get_object(), delete_object(), delete_bucket(). etc. > > There are things that doesn't work or need twisting. AA is one, which I worked around by using a macaroon token. Listing objects/buckets is another one because boto3 expect some kind of XML (probably specified in S3 protocol). > > XrdHTTP can identify access via boto3 api by checking the user-agent. The following is what I see in the log file: > > http_Protocol: rc:91 got hdr line: User-Agent: Boto3/1.12.49 Python/3.6.8 Linux/3.10.0-1160.49.1.el7.x86_64 Botocore/1.15.49 > > I don't know how far we can and should go along this route. I am assessing what is possible, and what is not. > > -- > Wei > > ________________________________________ > From: Fabrizio Furano <[log in to unmask]> > Sent: Monday, April 11, 2022 12:54 AM > To: Yang, Wei; xrootd-l > Subject: Re: Feature request on XrdHTTP > > Hi Wei, > > could you please be more specific about the requests that should get > an XML? > > I would prefer to keep the browsers working against XrdHttp > (which I think it's a great feature), so I would not just substitute > HTML with XML > > Can these requests that need XML be recognized through their headers? > > Also, boto3 uses the AWS protocol, while XrdHttp is ... HTTP. Does not > seem the perfect match to me. > > Maybe we can think at a plugin that provides basic S3 > compliance instead than adding tweaks and special cases? > > > Cheers > Fabrizio > > > > Il 09/04/2022 00:00, Yang, Wei ha scritto: >> I am playing with boto3 with XrdHTTP. After a few hack to pass a Macaroon to xrootd I can get boto3 to download a file from xrootd (I guess I can use scitoken too). But list_objects() and list_buckets() do not appreciate the HTML returned by the servers. They want XML. Could this be possible? >> >> -- >> Wei >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1