IFAIK, X=64.
Yes, RAL has a cache in the front to handle the load. But If I am not mistaken, the cache resides on RAL batch nodes and only will only serve the RAL batch nodes. So Raja, where are your streaming jobs running?
As Brian mentioned, even with a cache, the cache thrashing can still happens if the cache size is not big enough. In any case, it is best to start a discussion with RAL. I think RAL fully understand the usefulness of the cache.
regards,
--
Wei Yang | [log in to unmask]<mailto:[log in to unmask]> | 650-926-3338(O)
On 7/22/20, 7:30 AM, "Brian P Bockelman" <[log in to unmask]<mailto:[log in to unmask]>> wrote:
Hi @rajanandakumar<https://github.com/rajanandakumar> -
"Operation expired" simply means a timeout occurred before the xrootd server returned a response for a given operation (in this case, a vector read). In other words, the server didn't respond fast enough -- which can happen randomly, of course, as you may not know how busy the remote server is.
Unfortunately, this is possibly a side-effect of how RAL deploys its storage. IIRC (and do check up with them, my memory may be dated) the backend storage requires a minimum of X MB to be read (perhaps X=32?) for XRootD to service a read. So, vector read requests with many small reads may require up to the full file to be read out in one operation.
That is, if your read of 1024 chunks noted above is perfectly scattered throughout the file, then the backend may need to read out up to 5GB within the operation timeout to service the 1.7MB request.
RAL has historically addressed this through a caching layer - but that can be finicky to tune as large files can effectively permanently thrash the cache.
It's not clear that this is something that the XRootD team can solve. It might make sense to strike up a conversation with RAL - I know they've been studying this closely and may have some updates that invalidates my recollections above.
Brian
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/xrootd/xrootd/issues/1259#issuecomment-662487327>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHVGA2VIYXXWNJDCSOVUQLR43ZZHANCNFSM4PEQBI3Q>.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/xrootd/xrootd/issues/1259#issuecomment-662677815",
"url": "https://github.com/xrootd/xrootd/issues/1259#issuecomment-662677815",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1