Hi again,
To add to this - to some extend it seems xrootd actually stops putting
stuff on that disk (see the plot attached). One can see that disk b (the
one that runs full) stops having access times after a certain date. For
comparison 2 other disks are shown. But it seems that only happens when
the disk is 100% full.
Cheers,
Nikolai
On 2/28/23 15:36, Nikolai Hartmann wrote:
> Hi Matevz,
>
>>>> I still suspect something goes wrong with the disk selection for
>>>> placement of new files -- the full disk should simply not be
>>>> chosen over the other disks and thus stop filling up
> Does that mean it should stop filling up when it reaches the high
> watermark or at 100%?
> So even if i have the situation that all my access/modification times on
> that particular disk are rather new (since i cleared it after i saw the
> issue first) you think it shouldn't happen that this disk fills up
> completely?
>
> I can try to run a custom build in a new container if you give me the
> instructions. However it probably should be on top of v5.4.2 since we
> saw issues with newer versions (very slow transfers - didn't have time
> to look in detail into that).
>
> Cheers,
> Nikolai
>
> On 2/28/23 09:05, Matevz Tadel wrote:
>> Hi Andy,
>>
>> Yeah, the code also looked good to me ... but then, if a fuller disk
>> still gets
>> selected (with assumed default oss.alloc, fuzz=0), then perhaps it's
>> the cache
>> scan that assigns the partition a wrong free space value? I'm not
>> saying the
>> problem is in xrootd, maybe somebody else is playing tricks at the
>> system/container level?
>>
>> Matevz
>>
>> On 2/27/23 23:50, Andrew Hanushevsky wrote:
>>> Hi Matevz,
>>>
>>> It is XrdOssCache::Alloc() and I will gaurantee you that it will not
>>> choose a
>>> full disk if it has properly confiurted. The default is proper so I
>>> don't know
>>> if there have been any overrides here.
>>>
>>> Andy
>>>
>>>
>>> On Mon, 27 Feb 2023, Matevz Tadel wrote:
>>>
>>>> Hi Nikolai,
>>>>
>>>> I don't think this matters, xcache uses last file access time as
>>>> written in the
>>>> cinfo file.
>>>>
>>>> I still suspect something goes wrong with the disk selection for
>>>> placement of
>>>> new files -- the full disk should simply not be chosen over the
>>>> other disks and
>>>> thus stop filling up.
>>>>
>>>> Wiping the whole cache would help, at least for a while :) ... but
>>>> it would be
>>>> good to understand what is going on here. Would you be able to run
>>>> with a custom
>>>> build? Potentially we could just replace a single library to include
>>>> some
>>>> printouts.
>>>>
>>>> Andy, remind me please ... where is the code that does disk
>>>> selection? Is it
>>>> XrdOssCache::Alloc()?
>>>>
>>>> Cheers,
>>>> Matevz
>>>>
>>>> On 2/16/23 02:27, Nikolai Hartmann wrote:
>>>>> Hi again,
>>>>>
>>>>> Actually it seems to only change the "change" time (st_ctime)
>>>>>
>>>>> touch test
>>>>> stat test
>>>>> [...]
>>>>> Access: 2023-02-16 11:25:11.962804882 +0100
>>>>> Modify: 2023-02-16 11:25:11.962804882 +0100
>>>>> Change: 2023-02-16 11:25:11.962804882 +0100
>>>>> Birth: 2023-02-16 11:25:11.962804882 +0100
>>>>>
>>>>> chown xrootd test
>>>>> stat test
>>>>> [...]
>>>>> Access: 2023-02-16 11:25:11.962804882 +0100
>>>>> Modify: 2023-02-16 11:25:11.962804882 +0100
>>>>> Change: 2023-02-16 11:25:20.322843125 +0100
>>>>> Birth: 2023-02-16 11:25:11.962804882 +0100
>>>>>
>>>>> Does this play a role?
>>>>>
>>>>> Cheers,
>>>>> Nikolai
>>>>>
>>>>> On 2/16/23 11:18, Nikolai Hartmann wrote:
>>>>>> Hi Matevz (including xrootd list again which i forgot in the last
>>>>>> reply),
>>>>>>
>>>>>>> Well, if for some reason more new files are placed on a single disk,
>>>>>>> those files will be "newer" and purge would preferentially wipe data
>>>>>>> off other disks.
>>>>>> Mhhhh - then i have an idea how i may have triggered this. As
>>>>>> mentioned in my
>>>>>> first email the issue started after i updated my container image
>>>>>> and had to
>>>>>> change the xrootd user ids. This changes the Access time of the
>>>>>> files - if
>>>>>> that is used by xrootd to determine which files are newer than it
>>>>>> could just
>>>>>> be that the chown process walked this directory last and therefore
>>>>>> will purge
>>>>>> it last.
>>>>>> When i then deleted it when the disk ran full i made the problem
>>>>>> even worse
>>>>>> since now all the files that end up there are recently accessed.
>>>>>>
>>>>>> So deleting the whole cache should fix it?
>>>>>>
>>>>>> Cheers,
>>>>>> Nikolai
>>>>>>
>>>>>> On 2/16/23 10:50, Matevz Tadel wrote:
>>>>>>> Hi Andy, Nikolai,
>>>>>>>
>>>>>>> On 2/15/23 23:51, Andrew Hanushevsky wrote:
>>>>>>>> Hi Nikolai,
>>>>>>>>
>>>>>>>> Hmm, this sounds like an off by one problem in Xcache.
>>>>>>>
>>>>>>> How? XCache does not do disks, it just uses oss API to a pool.
>>>>>>>
>>>>>>>> The question is what is
>>>>>>>> the "one". It does seem that ity consistently does not purge
>>>>>>>> files from a
>>>>>>>> particular disk but then again it doesn't know about disks. So,
>>>>>>>> there is some
>>>>>>>> systematic issue that resolves to ignoring a disk. Matevz?
>>>>>>>
>>>>>>> Well, if for some reason more new files are placed on a single
>>>>>>> disk, those
>>>>>>> files
>>>>>>> will be "newer" and purge would preferentially wipe data off
>>>>>>> other disks.
>>>>>>>
>>>>>>> That's why I asked in the first email how disks are selected for
>>>>>>> new files and
>>>>>>> if we could inject some debug printouts there.
>>>>>>>
>>>>>>> Perhaps a coincidence, but the full disk is the one that is
>>>>>>> listed first by
>>>>>>> df.
>>>>>>>
>>>>>>> The docs say default for oss.alloc fuzz = 0 and that this "forces
>>>>>>> oss to
>>>>>>> always
>>>>>>> use the partition with the largest amount of free space" -- so
>>>>>>> the fuller one
>>>>>>> should never get selected for new files. And xcache does pass the
>>>>>>> appropriate
>>>>>>> oss.asize opaque parameter to open.
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://xrootd.slac.stanford.edu/doc/dev56/ofs_config.htm*_Toc116508676__;Iw!!Mih3wA!CPJXm6eN-2_hoD2H_DidLrJJIwTvYUTK7V8pRT64GhSwBlmFYugKLfTk2O6zoR2otc1TQNvfczttg_nl$
>>>>>>> Matevz
>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 16 Feb 2023, Nikolai Hartmann wrote:
>>>>>>>>
>>>>>>>>> Hi Andy,
>>>>>>>>>
>>>>>>>>> The behavior seems to be that it purges all the disks except
>>>>>>>>> one. After the
>>>>>>>>> other disks now again surpassed the threshold of 95% it seemed
>>>>>>>>> to trigger
>>>>>>>>> the
>>>>>>>>> cleanup and now i have this:
>>>>>>>>>
>>>>>>>>> Filesystem Type Size Used Avail Use%
>>>>>>>>> Mounted on
>>>>>>>>> /dev/sdb btrfs 5,5T 5,3T 215G 97%
>>>>>>>>> /srv/xcache/b
>>>>>>>>> /dev/sda btrfs 5,5T 5,0T 560G 90%
>>>>>>>>> /srv/xcache/a
>>>>>>>>> /dev/sdh btrfs 5,5T 4,9T 588G 90%
>>>>>>>>> /srv/xcache/h
>>>>>>>>> /dev/sdj btrfs 5,5T 4,9T 584G 90%
>>>>>>>>> /srv/xcache/j
>>>>>>>>> /dev/sdf btrfs 5,5T 4,9T 580G 90%
>>>>>>>>> /srv/xcache/f
>>>>>>>>> /dev/sdm btrfs 5,5T 5,0T 535G 91%
>>>>>>>>> /srv/xcache/m
>>>>>>>>> /dev/sdc btrfs 5,5T 5,0T 553G 91%
>>>>>>>>> /srv/xcache/c
>>>>>>>>> /dev/sdg btrfs 5,5T 4,9T 612G 90%
>>>>>>>>> /srv/xcache/g
>>>>>>>>> /dev/sdi btrfs 5,5T 4,9T 596G 90%
>>>>>>>>> /srv/xcache/i
>>>>>>>>> /dev/sdl btrfs 5,5T 5,0T 518G 91%
>>>>>>>>> /srv/xcache/l
>>>>>>>>> /dev/sdn btrfs 5,5T 4,9T 570G 90%
>>>>>>>>> /srv/xcache/n
>>>>>>>>> /dev/sde btrfs 5,5T 4,9T 593G 90%
>>>>>>>>> /srv/xcache/e
>>>>>>>>> /dev/sdk btrfs 5,5T 4,8T 677G 88%
>>>>>>>>> /srv/xcache/k
>>>>>>>>> /dev/sdd btrfs 5,5T 4,9T 602G 90%
>>>>>>>>> /srv/xcache/d
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Nikolai
>>>>>>>>>
>>>>>>>>> On 2/14/23 21:52, Andrew Hanushevsky wrote:
>>>>>>>>>> Hi Matevz & Nikolai,
>>>>>>>>>>
>>>>>>>>>> The allocation should favor the disk with the most free space
>>>>>>>>>> unless it's
>>>>>>>>>> atered using the oss.alloc directive:
>>>>>>>>>> https://urldefense.com/v3/__https://xrootd.slac.stanford.edu/doc/dev54/ofs_config.htm*_Toc89982400__;Iw!!Mih3wA!AsisYxoXis_6IdoiqK-BwdMsHfHTB41Z4-GEjaMqvO0PQHh6TqU8Sn79JUgDeJDLCvO63yQiG63Zu6syVA$
>>>>>>>>>> I don't think Nikolai specifies that and I don't think the pfc
>>>>>>>>>> alters it in
>>>>>>>>>> any way. So, I can't explain why we see that difference other
>>>>>>>>>> than via an
>>>>>>>>>> uneven purge.
>>>>>>>>>>
>>>>>>>>>> Andy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, 14 Feb 2023, Matevz Tadel wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Nikolai, Andy,
>>>>>>>>>>>
>>>>>>>>>>> I saw this a long time back, 2++ years. The thing is that
>>>>>>>>>>> xcache does oss
>>>>>>>>>>> df on
>>>>>>>>>>> the whole space and then deletes files without any knowledge
>>>>>>>>>>> of the
>>>>>>>>>>> usage on
>>>>>>>>>>> individual disks themselves. Placement of new files should
>>>>>>>>>>> prefer the more
>>>>>>>>>>> empty
>>>>>>>>>>> disks though, iirc.
>>>>>>>>>>>
>>>>>>>>>>> I remember asking Andy about how xcache could be made aware
>>>>>>>>>>> of individual
>>>>>>>>>>> disks
>>>>>>>>>>> and he prepared something for me but it got really
>>>>>>>>>>> complicated when I was
>>>>>>>>>>> trying
>>>>>>>>>>> to include this into the cache purge algorithm so I think I
>>>>>>>>>>> dropped this.
>>>>>>>>>>>
>>>>>>>>>>> Andy, could we sneak some debug printouts into oss new file disk
>>>>>>>>>>> selection to
>>>>>>>>>>> see if something is going wrong there?
>>>>>>>>>>>
>>>>>>>>>>> Nikolai, how fast does this happen? Is it a matter of days,
>>>>>>>>>>> ie, over many
>>>>>>>>>>> purge
>>>>>>>>>>> cycles? Is it always the same disk?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Matevz
>>>>>>>>>>>
>>>>>>>>>>> On 2/13/23 23:21, Nikolai Hartmann wrote:
>>>>>>>>>>>> Hi Andy,
>>>>>>>>>>>>
>>>>>>>>>>>> The config is the following:
>>>>>>>>>>>>
>>>>>>>>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/etp-computing/xcache-nspawn-lrz/-/blob/086e5ade5d27fc7d5ef59448c955523e453c091f/etc/xrootd/xcache.cfg__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVipwRIRYyR$
>>>>>>>>>>>> The directories for `oss.localroot` and `oss.space meta` are
>>>>>>>>>>>> on the
>>>>>>>>>>>> system
>>>>>>>>>>>> disk.
>>>>>>>>>>>> The `/srv/xcache/[a-m]` are individually mounted devices.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Nikolai
>>>>>>>>>>>>
>>>>>>>>>>>> On 2/14/23 00:34, Andrew Hanushevsky wrote:
>>>>>>>>>>>>> Hi Nikolai,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hmmm, no it seems you are the first one. Then again, not
>>>>>>>>>>>>> many people
>>>>>>>>>>>>> have a
>>>>>>>>>>>>> multi-disk setup. So, could you send a link to your config
>>>>>>>>>>>>> file? It
>>>>>>>>>>>>> might be
>>>>>>>>>>>>> the case that all of the metadata files wind up on the same
>>>>>>>>>>>>> disk and
>>>>>>>>>>>>> that is
>>>>>>>>>>>>> the source of the issue here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Andy
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, 13 Feb 2023, Nikolai Hartmann wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear xrootd-l,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm seeing the issue that one of the disks on one of our
>>>>>>>>>>>>>> xcache servers
>>>>>>>>>>>>>> fills
>>>>>>>>>>>>>> up disproportionally - that means it runs completely full
>>>>>>>>>>>>>> until i
>>>>>>>>>>>>>> get "no
>>>>>>>>>>>>>> space left on device" errors without xcache running
>>>>>>>>>>>>>> cleanup, while the
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>> disks still have plenty of space left. My current df output:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /dev/sdb btrfs 5,5T 5,2T 273G 96%
>>>>>>>>>>>>>> /srv/xcache/b
>>>>>>>>>>>>>> /dev/sda btrfs 5,5T 4,9T 584G 90%
>>>>>>>>>>>>>> /srv/xcache/a
>>>>>>>>>>>>>> /dev/sdh btrfs 5,5T 5,0T 562G 90%
>>>>>>>>>>>>>> /srv/xcache/h
>>>>>>>>>>>>>> /dev/sdj btrfs 5,5T 5,0T 551G 91%
>>>>>>>>>>>>>> /srv/xcache/j
>>>>>>>>>>>>>> /dev/sdf btrfs 5,5T 4,9T 579G 90%
>>>>>>>>>>>>>> /srv/xcache/f
>>>>>>>>>>>>>> [...]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you look at the first line you see that disk is 96%
>>>>>>>>>>>>>> full while the
>>>>>>>>>>>>>> others
>>>>>>>>>>>>>> are around 90%. The issue occurred the first time after i
>>>>>>>>>>>>>> built a new
>>>>>>>>>>>>>> container for running xrootd. That change involved
>>>>>>>>>>>>>> switching the
>>>>>>>>>>>>>> container
>>>>>>>>>>>>>> from centos7 to almalinux8 and changing the xrootd user id
>>>>>>>>>>>>>> (ran
>>>>>>>>>>>>>> chown and
>>>>>>>>>>>>>> chgrp afterwards on the cache directories which are bind
>>>>>>>>>>>>>> mounted). The
>>>>>>>>>>>>>> xrootd
>>>>>>>>>>>>>> version stayed the same (5.4.2). The high/low watermark
>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>> is the
>>>>>>>>>>>>>> following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pfc.diskusage 0.90 0.95
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I already tried clearing the misbehaving disk (after it
>>>>>>>>>>>>>> ran full to
>>>>>>>>>>>>>> 100%),
>>>>>>>>>>>>>> but now the issue is reappearing. Has anyone seen similar
>>>>>>>>>>>>>> issues or
>>>>>>>>>>>>>> does it
>>>>>>>>>>>>>> ring any bells for you?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One thing i checked is the size that xrootd reports in the
>>>>>>>>>>>>>> log for the
>>>>>>>>>>>>>> total
>>>>>>>>>>>>>> storage and that at least matches what i get when i sum
>>>>>>>>>>>>>> the entries
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>> `df`.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Nikolai
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ########################################################################
>>>>>>>>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe from the XROOTD-L list, click the following
>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVip_SnON6x$
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|
