XROOTD-L Archives

Support use of xrootd by HEP experiments

XROOTD-L@LISTSERV.SLAC.STANFORD.EDU

Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Fabrizio Furano <[log in to unmask]>
Reply To:
Support use of xrootd by HEP experiments <[log in to unmask]>
Date:
Tue, 13 Jul 2021 16:09:42 +0200
Content-Type:
text/plain
Parts/Attachments:
text/plain (164 lines)
Hi,

 quite a complex setup that I am not sure I understand.

 What are you using as a cache?
 Is XrdCeph speaking xrootd or http in your case?
 What is xrdcp talking to? To the cache or to the backend XrdCeph?
 What is the advantage of putting a cache in front of XrdCeph?

 My suspicion is that some components of these is opening
a new connection for every chunk requested, this would
explain the effect that you see. Of course I may be wrong,
and surely you can find the final answer in the debug logs.

 If this is true, then putting unreasonably sized internal buffers
would just be a workaround for a more important problem to solve.

Fabrizio




Il 13/07/2021 13:53, James William Walder ha scritto:
> Hi Fabrizio, 
>   Thanks for the response. 
> Our Gateways are configured with a memory-cache proxy with a server on
> the same node, configured using XrdCeph to access the Ceph/rados storage
> (as object store). 
> Ceph is configured with a 64MiB stripe size.
> 
> There are some outstanding concerns on optimal configuration of a
> memcache, which may or may-not be related.
> 
> As a figure-of-merit comparison (take the relative comparison of numbers
> more seriously than absolute),
>      using root:// with an xrdcp download, and varying the client
> XRD_CPCHUNKSIZE speeds can range from:
> 
> XRD_CPCHUNKSIZE  |  Download speed (root://)
> 1 MiB ~  4 MB/s
> 4 MiB ~ 11MiB/s
> 8 MiB ~ 20 MB/s
> 16 MiB ~ 35 MB/s
> 32MiB ~ 50 MB/s
> 64MiB ~ 60 MB/s
> 
> My feeling was, if http is restricted to 1MiB (or lower) transfer sizes,
> then we’re not running optimally. 
> I’m happy to believe there are other areas for improvement/understanding
> (e.g. the men-cache, which Tom Byrnne brought up in this mail list), 
> but trying to do a direct comparison between root and webdav seemed like
> a good start.
> 
> Thanks in advance,
> James
> 
> 
> 
> 
> 
> 
> 
> 
>> On 12 Jul 2021, at 14:46, Fabrizio Furano <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>>
>> Hi,
>>
>> of course you are free to experiment, however my feeling is that
>> this may just be an attempt to hide some other issue elsewhere.
>> Which system do you think is sensitive to this size?
>>
>> Fabrizio
>>
>> On 09.07.21 22:43, James William Walder wrote:
>>> Hi,
>>>   I’d like to investigate various ‘block size’ variations in TPC-HTTP
>>> (especially writes); 
>>> (by which I mean the size of the write that is passed to the server from
>>> the proxy, I hope that makes sense … ) 
>>>
>>> By default, I think writes are written at 1MiB boundaries.
>>> From the code, I’m hoping I could make the following changes (see diff
>>> below), but it’s not clear if there are some subtle effects to be aware
>>> of, or if I’ve missed some obvious problem (i.e. like making sure they
>>> are at least written at sensible sizes). 
>>>
>>> The motivation is that I’m observing slow transfers in http-tpc
>>> (relative to root for example), and would like to test the effect of
>>> ‘block size’ changes, for which we are quite sensitive to.
>>> Any thoughts would be appreciated.
>>>
>>> Thanks in advance,
>>> James
>>>
>>>
>>> (Based on 5.3.0-1) 
>>>
>>> diff --git a/src/XrdTpc/XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/> <http://XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/>>
>>> b/src/XrdTpc/XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/> <http://XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/>>
>>> index 76c7d6e..c2ff92d 100644
>>> --- a/src/XrdTpc/XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/> <http://XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/>>
>>> +++ b/src/XrdTpc/XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/> <http://XrdTpcStream.cc
>>> <http://xrdtpcstream.cc/>>
>>> @@ -80,7 +80,7 @@ Stream::Write(off_t offset, const char *buf, size_t
>>> size, bool force)
>>>      // If this is write is appending to the stream and
>>>      // MB-aligned, then we write it to disk; otherwise, the
>>>      // data will be buffered.
>>> -    if (offset == m_offset && (force || (size && !(size %
>>> (1024*1024))))) {
>>> +    if (offset == m_offset && (force || (size && !(size %
>>> (16*1024*1024))))) {
>>>          retval = WriteImpl(offset, buf, size);
>>>          bytes_accepted = retval;
>>>              // On failure, we don't care about flushing buffers from
>>> memory —
>>>
>>> diff --git a/src/XrdTpc/XrdTpcTPC.cc
>>> <http://xrdtpctpc.cc/> <http://XrdTpcTPC.cc <http://xrdtpctpc.cc/>>
>>> b/src/XrdTpc/XrdTpcTPC.cc <http://xrdtpctpc.cc/> <http://XrdTpcTPC.cc
>>> <http://xrdtpctpc.cc/>>
>>> index d800dfa..fff4a1e 100644
>>> --- a/src/XrdTpc/XrdTpcTPC.cc
>>> <http://xrdtpctpc.cc/> <http://XrdTpcTPC.cc <http://xrdtpctpc.cc/>>
>>> +++ b/src/XrdTpc/XrdTpcTPC.cc
>>> <http://xrdtpctpc.cc/> <http://XrdTpcTPC.cc <http://xrdtpctpc.cc/>>
>>> @@ -26,7 +26,7 @@ using namespace TPC;
>>>  uint64_t TPCHandler::m_monid{0};
>>>  int TPCHandler::m_marker_period = 5;
>>>  size_t TPCHandler::m_block_size = 16*1024*1024;
>>> -size_t TPCHandler::m_small_block_size = 1*1024*1024;
>>> +size_t TPCHandler::m_small_block_size = 16*1024*1024;
>>>  XrdSysMutex TPCHandler::m_monid_mutex;
>>>
>>>  
>>>
>>>  XrdVERSIONINFO(XrdHttpGetExtHandler, HttpTPC);
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>> <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1>
>>> <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>> <https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1>>
> 

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

ATOM RSS1 RSS2