So, I have an alternative explanation. Let's say that the initial
allocation is correct. The files are spread so as to have an equal load.
Note that xrootd doesn't know how big the file will be unless someone
tells it. If no one tells it then it uses a configurable default amount.
That space stays "in use" to avoid placing even more files on that disk
but it doesn' stay that way forever. After a configurable amount of time
the space allocated tracks the actual space in use. However, Xcache is
free to keep writing to that file past the last calculated amount of free
space. If that file is unusually active, that is a lot of data has been
fetched for it, then that disk will appear fuller than others.
So, the bottom line here is that the nature of how space is used in
Xcache creates a situation where it's not really possible to fully balance
all of the disks. The balance will always be approximate.
Andy
On Tue, 28 Feb 2023, Nikolai Hartmann wrote:
> Hi again,
>
> To add to this - to some extend it seems xrootd actually stops putting stuff
> on that disk (see the plot attached). One can see that disk b (the one that
> runs full) stops having access times after a certain date. For comparison 2
> other disks are shown. But it seems that only happens when the disk is 100%
> full.
>
> Cheers,
> Nikolai
>
> On 2/28/23 15:36, Nikolai Hartmann wrote:
>> Hi Matevz,
>>
>>>>> I still suspect something goes wrong with the disk selection for
>>>>> placement of new files -- the full disk should simply not be
>>>>> chosen over the other disks and thus stop filling up
>> Does that mean it should stop filling up when it reaches the high watermark
>> or at 100%?
>> So even if i have the situation that all my access/modification times on
>> that particular disk are rather new (since i cleared it after i saw the
>> issue first) you think it shouldn't happen that this disk fills up
>> completely?
>>
>> I can try to run a custom build in a new container if you give me the
>> instructions. However it probably should be on top of v5.4.2 since we saw
>> issues with newer versions (very slow transfers - didn't have time to look
>> in detail into that).
>>
>> Cheers,
>> Nikolai
>>
>> On 2/28/23 09:05, Matevz Tadel wrote:
>>> Hi Andy,
>>>
>>> Yeah, the code also looked good to me ... but then, if a fuller disk still
>>> gets
>>> selected (with assumed default oss.alloc, fuzz=0), then perhaps it's the
>>> cache
>>> scan that assigns the partition a wrong free space value? I'm not saying
>>> the
>>> problem is in xrootd, maybe somebody else is playing tricks at the
>>> system/container level?
>>>
>>> Matevz
>>>
>>> On 2/27/23 23:50, Andrew Hanushevsky wrote:
>>>> Hi Matevz,
>>>>
>>>> It is XrdOssCache::Alloc() and I will gaurantee you that it will not
>>>> choose a
>>>> full disk if it has properly confiurted. The default is proper so I don't
>>>> know
>>>> if there have been any overrides here.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On Mon, 27 Feb 2023, Matevz Tadel wrote:
>>>>
>>>>> Hi Nikolai,
>>>>>
>>>>> I don't think this matters, xcache uses last file access time as written
>>>>> in the
>>>>> cinfo file.
>>>>>
>>>>> I still suspect something goes wrong with the disk selection for
>>>>> placement of
>>>>> new files -- the full disk should simply not be chosen over the other
>>>>> disks and
>>>>> thus stop filling up.
>>>>>
>>>>> Wiping the whole cache would help, at least for a while :) ... but it
>>>>> would be
>>>>> good to understand what is going on here. Would you be able to run with
>>>>> a custom
>>>>> build? Potentially we could just replace a single library to include
>>>>> some
>>>>> printouts.
>>>>>
>>>>> Andy, remind me please ... where is the code that does disk selection?
>>>>> Is it
>>>>> XrdOssCache::Alloc()?
>>>>>
>>>>> Cheers,
>>>>> Matevz
>>>>>
>>>>> On 2/16/23 02:27, Nikolai Hartmann wrote:
>>>>>> Hi again,
>>>>>>
>>>>>> Actually it seems to only change the "change" time (st_ctime)
>>>>>>
>>>>>> touch test
>>>>>> stat test
>>>>>> [...]
>>>>>> Access: 2023-02-16 11:25:11.962804882 +0100
>>>>>> Modify: 2023-02-16 11:25:11.962804882 +0100
>>>>>> Change: 2023-02-16 11:25:11.962804882 +0100
>>>>>> Birth: 2023-02-16 11:25:11.962804882 +0100
>>>>>>
>>>>>> chown xrootd test
>>>>>> stat test
>>>>>> [...]
>>>>>> Access: 2023-02-16 11:25:11.962804882 +0100
>>>>>> Modify: 2023-02-16 11:25:11.962804882 +0100
>>>>>> Change: 2023-02-16 11:25:20.322843125 +0100
>>>>>> Birth: 2023-02-16 11:25:11.962804882 +0100
>>>>>>
>>>>>> Does this play a role?
>>>>>>
>>>>>> Cheers,
>>>>>> Nikolai
>>>>>>
>>>>>> On 2/16/23 11:18, Nikolai Hartmann wrote:
>>>>>>> Hi Matevz (including xrootd list again which i forgot in the last
>>>>>>> reply),
>>>>>>>
>>>>>>>> Well, if for some reason more new files are placed on a single disk,
>>>>>>>> those files will be "newer" and purge would preferentially wipe data
>>>>>>>> off other disks.
>>>>>>> Mhhhh - then i have an idea how i may have triggered this. As
>>>>>>> mentioned in my
>>>>>>> first email the issue started after i updated my container image and
>>>>>>> had to
>>>>>>> change the xrootd user ids. This changes the Access time of the files
>>>>>>> - if
>>>>>>> that is used by xrootd to determine which files are newer than it
>>>>>>> could just
>>>>>>> be that the chown process walked this directory last and therefore
>>>>>>> will purge
>>>>>>> it last.
>>>>>>> When i then deleted it when the disk ran full i made the problem even
>>>>>>> worse
>>>>>>> since now all the files that end up there are recently accessed.
>>>>>>>
>>>>>>> So deleting the whole cache should fix it?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Nikolai
>>>>>>>
>>>>>>> On 2/16/23 10:50, Matevz Tadel wrote:
>>>>>>>> Hi Andy, Nikolai,
>>>>>>>>
>>>>>>>> On 2/15/23 23:51, Andrew Hanushevsky wrote:
>>>>>>>>> Hi Nikolai,
>>>>>>>>>
>>>>>>>>> Hmm, this sounds like an off by one problem in Xcache.
>>>>>>>>
>>>>>>>> How? XCache does not do disks, it just uses oss API to a pool.
>>>>>>>>
>>>>>>>>> The question is what is
>>>>>>>>> the "one". It does seem that ity consistently does not purge files
>>>>>>>>> from a
>>>>>>>>> particular disk but then again it doesn't know about disks. So,
>>>>>>>>> there is some
>>>>>>>>> systematic issue that resolves to ignoring a disk. Matevz?
>>>>>>>>
>>>>>>>> Well, if for some reason more new files are placed on a single disk,
>>>>>>>> those
>>>>>>>> files
>>>>>>>> will be "newer" and purge would preferentially wipe data off other
>>>>>>>> disks.
>>>>>>>>
>>>>>>>> That's why I asked in the first email how disks are selected for new
>>>>>>>> files and
>>>>>>>> if we could inject some debug printouts there.
>>>>>>>>
>>>>>>>> Perhaps a coincidence, but the full disk is the one that is listed
>>>>>>>> first by
>>>>>>>> df.
>>>>>>>>
>>>>>>>> The docs say default for oss.alloc fuzz = 0 and that this "forces oss
>>>>>>>> to
>>>>>>>> always
>>>>>>>> use the partition with the largest amount of free space" -- so the
>>>>>>>> fuller one
>>>>>>>> should never get selected for new files. And xcache does pass the
>>>>>>>> appropriate
>>>>>>>> oss.asize opaque parameter to open.
>>>>>>>>
>>>>>>>> https://urldefense.com/v3/__https://xrootd.slac.stanford.edu/doc/dev56/ofs_config.htm*_Toc116508676__;Iw!!Mih3wA!CPJXm6eN-2_hoD2H_DidLrJJIwTvYUTK7V8pRT64GhSwBlmFYugKLfTk2O6zoR2otc1TQNvfczttg_nl$
>>>>>>>> Matevz
>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 16 Feb 2023, Nikolai Hartmann wrote:
>>>>>>>>>
>>>>>>>>>> Hi Andy,
>>>>>>>>>>
>>>>>>>>>> The behavior seems to be that it purges all the disks except one.
>>>>>>>>>> After the
>>>>>>>>>> other disks now again surpassed the threshold of 95% it seemed to
>>>>>>>>>> trigger
>>>>>>>>>> the
>>>>>>>>>> cleanup and now i have this:
>>>>>>>>>>
>>>>>>>>>> Filesystem Type Size Used Avail Use% Mounted
>>>>>>>>>> on
>>>>>>>>>> /dev/sdb btrfs 5,5T 5,3T 215G 97%
>>>>>>>>>> /srv/xcache/b
>>>>>>>>>> /dev/sda btrfs 5,5T 5,0T 560G 90%
>>>>>>>>>> /srv/xcache/a
>>>>>>>>>> /dev/sdh btrfs 5,5T 4,9T 588G 90%
>>>>>>>>>> /srv/xcache/h
>>>>>>>>>> /dev/sdj btrfs 5,5T 4,9T 584G 90%
>>>>>>>>>> /srv/xcache/j
>>>>>>>>>> /dev/sdf btrfs 5,5T 4,9T 580G 90%
>>>>>>>>>> /srv/xcache/f
>>>>>>>>>> /dev/sdm btrfs 5,5T 5,0T 535G 91%
>>>>>>>>>> /srv/xcache/m
>>>>>>>>>> /dev/sdc btrfs 5,5T 5,0T 553G 91%
>>>>>>>>>> /srv/xcache/c
>>>>>>>>>> /dev/sdg btrfs 5,5T 4,9T 612G 90%
>>>>>>>>>> /srv/xcache/g
>>>>>>>>>> /dev/sdi btrfs 5,5T 4,9T 596G 90%
>>>>>>>>>> /srv/xcache/i
>>>>>>>>>> /dev/sdl btrfs 5,5T 5,0T 518G 91%
>>>>>>>>>> /srv/xcache/l
>>>>>>>>>> /dev/sdn btrfs 5,5T 4,9T 570G 90%
>>>>>>>>>> /srv/xcache/n
>>>>>>>>>> /dev/sde btrfs 5,5T 4,9T 593G 90%
>>>>>>>>>> /srv/xcache/e
>>>>>>>>>> /dev/sdk btrfs 5,5T 4,8T 677G 88%
>>>>>>>>>> /srv/xcache/k
>>>>>>>>>> /dev/sdd btrfs 5,5T 4,9T 602G 90%
>>>>>>>>>> /srv/xcache/d
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Nikolai
>>>>>>>>>>
>>>>>>>>>> On 2/14/23 21:52, Andrew Hanushevsky wrote:
>>>>>>>>>>> Hi Matevz & Nikolai,
>>>>>>>>>>>
>>>>>>>>>>> The allocation should favor the disk with the most free space
>>>>>>>>>>> unless it's
>>>>>>>>>>> atered using the oss.alloc directive:
>>>>>>>>>>> https://urldefense.com/v3/__https://xrootd.slac.stanford.edu/doc/dev54/ofs_config.htm*_Toc89982400__;Iw!!Mih3wA!AsisYxoXis_6IdoiqK-BwdMsHfHTB41Z4-GEjaMqvO0PQHh6TqU8Sn79JUgDeJDLCvO63yQiG63Zu6syVA$
>>>>>>>>>>> I don't think Nikolai specifies that and I don't think the pfc
>>>>>>>>>>> alters it in
>>>>>>>>>>> any way. So, I can't explain why we see that difference other than
>>>>>>>>>>> via an
>>>>>>>>>>> uneven purge.
>>>>>>>>>>>
>>>>>>>>>>> Andy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, 14 Feb 2023, Matevz Tadel wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Nikolai, Andy,
>>>>>>>>>>>>
>>>>>>>>>>>> I saw this a long time back, 2++ years. The thing is that xcache
>>>>>>>>>>>> does oss
>>>>>>>>>>>> df on
>>>>>>>>>>>> the whole space and then deletes files without any knowledge of
>>>>>>>>>>>> the
>>>>>>>>>>>> usage on
>>>>>>>>>>>> individual disks themselves. Placement of new files should prefer
>>>>>>>>>>>> the more
>>>>>>>>>>>> empty
>>>>>>>>>>>> disks though, iirc.
>>>>>>>>>>>>
>>>>>>>>>>>> I remember asking Andy about how xcache could be made aware of
>>>>>>>>>>>> individual
>>>>>>>>>>>> disks
>>>>>>>>>>>> and he prepared something for me but it got really complicated
>>>>>>>>>>>> when I was
>>>>>>>>>>>> trying
>>>>>>>>>>>> to include this into the cache purge algorithm so I think I
>>>>>>>>>>>> dropped this.
>>>>>>>>>>>>
>>>>>>>>>>>> Andy, could we sneak some debug printouts into oss new file disk
>>>>>>>>>>>> selection to
>>>>>>>>>>>> see if something is going wrong there?
>>>>>>>>>>>>
>>>>>>>>>>>> Nikolai, how fast does this happen? Is it a matter of days, ie,
>>>>>>>>>>>> over many
>>>>>>>>>>>> purge
>>>>>>>>>>>> cycles? Is it always the same disk?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Matevz
>>>>>>>>>>>>
>>>>>>>>>>>> On 2/13/23 23:21, Nikolai Hartmann wrote:
>>>>>>>>>>>>> Hi Andy,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The config is the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/etp-computing/xcache-nspawn-lrz/-/blob/086e5ade5d27fc7d5ef59448c955523e453c091f/etc/xrootd/xcache.cfg__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVipwRIRYyR$
>>>>>>>>>>>>> The directories for `oss.localroot` and `oss.space meta` are on
>>>>>>>>>>>>> the
>>>>>>>>>>>>> system
>>>>>>>>>>>>> disk.
>>>>>>>>>>>>> The `/srv/xcache/[a-m]` are individually mounted devices.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Nikolai
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2/14/23 00:34, Andrew Hanushevsky wrote:
>>>>>>>>>>>>>> Hi Nikolai,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hmmm, no it seems you are the first one. Then again, not many
>>>>>>>>>>>>>> people
>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>> multi-disk setup. So, could you send a link to your config
>>>>>>>>>>>>>> file? It
>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>> the case that all of the metadata files wind up on the same
>>>>>>>>>>>>>> disk and
>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>> the source of the issue here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Andy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, 13 Feb 2023, Nikolai Hartmann wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dear xrootd-l,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm seeing the issue that one of the disks on one of our
>>>>>>>>>>>>>>> xcache servers
>>>>>>>>>>>>>>> fills
>>>>>>>>>>>>>>> up disproportionally - that means it runs completely full
>>>>>>>>>>>>>>> until i
>>>>>>>>>>>>>>> get "no
>>>>>>>>>>>>>>> space left on device" errors without xcache running cleanup,
>>>>>>>>>>>>>>> while the
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>> disks still have plenty of space left. My current df output:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /dev/sdb btrfs 5,5T 5,2T 273G 96%
>>>>>>>>>>>>>>> /srv/xcache/b
>>>>>>>>>>>>>>> /dev/sda btrfs 5,5T 4,9T 584G 90%
>>>>>>>>>>>>>>> /srv/xcache/a
>>>>>>>>>>>>>>> /dev/sdh btrfs 5,5T 5,0T 562G 90%
>>>>>>>>>>>>>>> /srv/xcache/h
>>>>>>>>>>>>>>> /dev/sdj btrfs 5,5T 5,0T 551G 91%
>>>>>>>>>>>>>>> /srv/xcache/j
>>>>>>>>>>>>>>> /dev/sdf btrfs 5,5T 4,9T 579G 90%
>>>>>>>>>>>>>>> /srv/xcache/f
>>>>>>>>>>>>>>> [...]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you look at the first line you see that disk is 96% full
>>>>>>>>>>>>>>> while the
>>>>>>>>>>>>>>> others
>>>>>>>>>>>>>>> are around 90%. The issue occurred the first time after i
>>>>>>>>>>>>>>> built a new
>>>>>>>>>>>>>>> container for running xrootd. That change involved switching
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> container
>>>>>>>>>>>>>>> from centos7 to almalinux8 and changing the xrootd user id
>>>>>>>>>>>>>>> (ran
>>>>>>>>>>>>>>> chown and
>>>>>>>>>>>>>>> chgrp afterwards on the cache directories which are bind
>>>>>>>>>>>>>>> mounted). The
>>>>>>>>>>>>>>> xrootd
>>>>>>>>>>>>>>> version stayed the same (5.4.2). The high/low watermark
>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>> is the
>>>>>>>>>>>>>>> following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pfc.diskusage 0.90 0.95
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I already tried clearing the misbehaving disk (after it ran
>>>>>>>>>>>>>>> full to
>>>>>>>>>>>>>>> 100%),
>>>>>>>>>>>>>>> but now the issue is reappearing. Has anyone seen similar
>>>>>>>>>>>>>>> issues or
>>>>>>>>>>>>>>> does it
>>>>>>>>>>>>>>> ring any bells for you?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One thing i checked is the size that xrootd reports in the log
>>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>>> total
>>>>>>>>>>>>>>> storage and that at least matches what i get when i sum the
>>>>>>>>>>>>>>> entries
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> `df`.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Nikolai
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ########################################################################
>>>>>>>>>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To unsubscribe from the XROOTD-L list, click the following
>>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVip_SnON6x$
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|