Hi,
AFAIU (Brian) you are proposing a buffering layer that
accumulates sequential writes before flushing to the fs layer.
I think that the behaviour of such a buffer would be
dependent on the needs of the underlying fs implementation, and
in this case (correct me please if I am wrong) the underlying implementation
(and the only one that would need it) is XrdCeph. So I would
implement it in XrdCeph as a private write buffer...
f
On 10/12/21 11:09 PM, xrootd-dev wrote:
> It's not clear what is meant by "filesystem layer". I do not like mucking
> around a generic interface to simply satisfy one protocol that may not
> staisfy another. So, I'd like to keep the complexity level to a minimum.
> We have a hard enough time with http as it is.
>
> On Tue, 12 Oct 2021, Brian P Bockelman wrote:
>
>> Hi James -- I think this sounds like a great idea.
>>
>> For TPC, you found the correct location for increasing the buffer size. Note `TPCHandler::m_block_size` is the size of the
> HTTP request generated when HTTP-TPC is doing multistreams and must be a multiple of the block size (otherwise you may hit
> deadlocks). My recommendation is to make it no smaller than 16MB.
>>
>> I never did much exploration into optimal block sizes; without it, `libcurl` would generate extremely small writes (16KB or
> smaller, depending on how often you poll the TCP socket) and the Ceph RADOS integration would outright fail with unaligned
> writes. So, I'd welcome your work!
>>
>> The hard decision point is the buffering in XrdHttp. Right now, the size of the buffer for PUT appears to simply be whatever
> comes off the socket. You _may_ put the buffer in the XrdHttp layer ... or it might make more sense at the filesystem layer. We
> already "paid the price" in XrdTpc in terms of code complexity but it's not immediately clear the same decision makes sense for
> XrdHttp.
>>
>> --
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly or view it on GitHub:
>> https://github.com/xrootd/xrootd/issues/1532#issuecomment-941288163
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/xrootd/xrootd/issues/1532#issuecomment-941560859>, or
> unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJBUTYPWHRLQUOEGXDQ7X3UGSPYBANCNFSM5F2J2QJA>.
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/xrootd/xrootd/issues/1532#issuecomment-941997652",
"url": "https://github.com/xrootd/xrootd/issues/1532#issuecomment-941997652",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
