Sorry for being terse on the original comment -- sometimes I have to squeeze these thoughts between other duties.
My insight is the problem is because we try and execute the HTTP response as part of a single vector IO request to the XRootD layer. There's no need for that! For example, when we read the whole file, we simply repeat 128KB (non-vector) read requests to XRootD until we get to the end of the file. There's no need to have a 1-to-1 relationship; it can be 1-to-N.
We can do the same thing here:
1. Based on the remaining, unfilled HTTP range requests, create a vector read that is no more than 1024 xrootd requests. You can even keep the current size limits of 128kb (or pick any arbitrary size limit that Andy thinks would align well).
- May be a good idea to keep a queue-like structure of unfilled range request so you're simply popping things off the queue.
3. Issue the vector read, push the data down the socket.
4. If there's remaining data requested by the client, go back to 1. Repeat until all requested data is sent.
@ccaffy, if you want to tackle this, I'd suggest taking a look at https://github.com/xrootd/xrootd/pull/1957/commits/36df693600a5bb9d98a2421ad239115aefdf9dad; ignore the new feature that was working on but I'd suggest copying the cleanup of the state machine into a standalone commit, then try to build the above algorithm on the cleaner state machine.
--
Reply to this email directly or view it on GitHub:
https://github.com/xrootd/xrootd/issues/1976#issuecomment-1579092442
You are receiving this because you are subscribed to this thread.
Message ID: <[log in to unmask]>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
