 |
 |
Dear Sam, On 1/20/21 5:44 AM, Sam Skipsey wrote: > Dear xrootd-l: > > I'm currently seeing a hard to understand error with an internal Xrootd proxy cache. > > Normally, the cache has functioned as expected - purging when it gets above > highWatermark [and returning XrdFileCache_File: error File::Open() Create failed > , err_code=28, err_str=No space left on device (and forwarding requests) while > it does so, until it frees up below lowWatermark again. This is a bit strange ... on File::Open XCache does not check actual disk usage -- so the error is genuine, coming from the XrdOss layer. Disk usage vs. high watermark is only checked in the purge thread ran periodically. If usage is above, the purge scan is performed and corresponding amount of data purged. > The cache uses xrootd spaces to manage its backend disk storage, which is > distributed across 6 volumes, all of which are SSDs. > > However, we currently see the same behaviour on the cache today, with all 6 > filesystems in the cache only ~70% full [our high and low watermarks are 95% and > 90%]; and this has been ongoing for much longer than the refresh time for the > cache's metrics. > > I can write to the filesystems involved, so it isn't a problem with the > underlying storage. > > We are on 5.0.3 . > > Does anyone have any suggestions? The purpose of this cache is primarily to > smooth and consolidate load, so it forwarding most of its requests is, whilst > bearable in the short term, not really good for our performance or reliability. We have to understand why this happens :) Would you mind running with 5.1.0-rc5 that has just been released? Andy, what would be the simplest way to reproduce this? Running standalone mini-server with localroot on a ssd partition and then writing in with xrdcp? Are there some tracing options that would help? Best, Matevz > Yours, > > Sam > > > -------------------------------------------------------------------------------- > > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > <https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!Vo-7lXsDpPrxg9kGWsQ64ELB_iKG3mm717MbGNArYT5sLW0_YdeuXSUGbK5Ia1kf5jAD$> > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1