Print

Print


Hi Michal,
Thanks for delving into the class hierarchy for us! The approach of implementing ReadV in the Ceph plugin tallies with the discussion we had with Andy and Brian a while ago. For reference, the design document (it's a WiP) for the RAL implementation is at
https://docs.google.com/document/d/1VxMKULKMd6PGNfSsQStWW8BHvbhM4yTYDc6jo5GByxw/edit?usp=sharing
(attached in case the Google doc isn't readable). Whilst George and I are confident that we have an initial design suitable to make into a prototype, any comments are of course welcome.

Best Regards,
Ian.

________________________________
From: Michal Simon <[log in to unmask]>
Sent: 11 December 2020 11:17
To: Patargias, George (STFC,RAL,SC) <[log in to unmask]>
Cc: Bockelman, Brian <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; Johnson, Ian (STFC,RAL,SC) <[log in to unmask]>
Subject: Re: Handling of vector-reads in the the XRootD

Hi George,

Well, if there is a client application that uses XrdCl::File::VectorRead this will be translated into a kXR_readv request and send to the server.
(If you want to trigger the server to do a readv just write a dummy app that does File::Open + File::ReadV + File::Close)

The request will be handled by the server in here:
https://github.com/xrootd/xrootd/blob/master/src/XrdXrootd/XrdXrootdXeq.cc#L2393

The server will use the OFS plug-in to do readv:
https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfs.cc#L1099

The OFS plug-in in turn will use the OSS plug-in to do the ready:
https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfs.cc#L1116

The OSS layer is implemented by the Ceph plug-in, however it does not define the ReadV method, so this actually calls the implementation from the base class:
https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfs.cc#L1116

and this one is just looping over the iovec and calls a synchronous read for each buffer in the vector:
https://github.com/xrootd/xrootd/blob/master/src/XrdOss/XrdOss.cc#L257-L266

I think that if would just overload the ReadV routine in the ceph plugin and would use the async read (int Read(XrdSfsAio *aoip)
instead, this would do the trick.


I hope that helps!


Cheers,
Michal

On 11 Dec 2020, at 11:38, Patargias, George (STFC,RAL,SC) <[log in to unmask]<mailto:[log in to unmask]>> wrote:

Hi Brian,

Thanks for the reply.

Indeed, I couldn't find any flag from the client side that invokes a vector read, so I meant the server side.

Basically, I am trying to figure out how XrdCl::File::VectorRead is invoked and what is the subsequent sequence of calls that results in the construction of the XrdOucIOVec structure (XrdOuc/XrdOucIOVec.hh) that will be passed as
an argument to the ReadV(XrdOucIOVec *readV, int rdvcnt) function in XrdOss/XrdOss.cc<http://xrdoss.cc/>.

So, any clues on the relation between XrdCl::File::VectorRead  and XrdOssDF::ReadV would be very useful.

Best,

George




________________________________
From: Bockelman, Brian <[log in to unmask]<mailto:[log in to unmask]>>
Sent: 10 December 2020 22:57
To: Patargias, George (STFC,RAL,SC) <[log in to unmask]<mailto:[log in to unmask]>>
Cc: [log in to unmask]<mailto:[log in to unmask]> <[log in to unmask]<mailto:[log in to unmask]>>; [log in to unmask]<mailto:[log in to unmask]> <[log in to unmask]<mailto:[log in to unmask]>>; Johnson, Ian (STFC,RAL,SC) <[log in to unmask]<mailto:[log in to unmask]>>
Subject: Re: Handling of vector-reads in the the XRootD

Hi George,

Did you mean in the client or server side?

In the client, XrdCl::File::VectorRead is what is invoked by various applications (e.g., ROOT) in order to trigger a vector read.

I don't believe xrdcp ever invokes a vector read.

Brian

> On Dec 10, 2020, at 11:35 AM, Patargias, George (STFC,RAL,SC) <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>
> Hello,
>
> Me and Ian are working in RAL on the implementation of vector reads in the XRootD Ceph plugin.
>
> Can you please point us to the part of the XRootD code (XrdCl or...) that groups many individual small reads from the same
> file into a single vector (XrdOucIOVec) which is eventually passed to the ReadV(XrdOucIOVec *readV, int rdvcnt) function
> in XrdOss/XrdOss.cc<http://XrdOss.cc>?
>
> Is this vectorisation enabled by default in the case of many reads from the same file (didn't see any xrdcp flag that serves this purpose)?
>
> Many thanks,
>
> George Patargias
> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.
>
> Use REPLY-ALL to reply to list
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1


________________________________
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1


########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1