 |
 |
Hi Matevz (including xrootd list again which i forgot in the last reply), > Well, if for some reason more new files are placed on a single disk, > those files will be "newer" and purge would preferentially wipe data > off other disks. Mhhhh - then i have an idea how i may have triggered this. As mentioned in my first email the issue started after i updated my container image and had to change the xrootd user ids. This changes the Access time of the files - if that is used by xrootd to determine which files are newer than it could just be that the chown process walked this directory last and therefore will purge it last. When i then deleted it when the disk ran full i made the problem even worse since now all the files that end up there are recently accessed. So deleting the whole cache should fix it? Cheers, Nikolai On 2/16/23 10:50, Matevz Tadel wrote: > Hi Andy, Nikolai, > > On 2/15/23 23:51, Andrew Hanushevsky wrote: >> Hi Nikolai, >> >> Hmm, this sounds like an off by one problem in Xcache. > > How? XCache does not do disks, it just uses oss API to a pool. > >> The question is what is >> the "one". It does seem that ity consistently does not purge files from a >> particular disk but then again it doesn't know about disks. So, there is some >> systematic issue that resolves to ignoring a disk. Matevz? > > Well, if for some reason more new files are placed on a single disk, those files > will be "newer" and purge would preferentially wipe data off other disks. > > That's why I asked in the first email how disks are selected for new files and > if we could inject some debug printouts there. > > Perhaps a coincidence, but the full disk is the one that is listed first by df. > > The docs say default for oss.alloc fuzz = 0 and that this "forces oss to always > use the partition with the largest amount of free space" -- so the fuller one > should never get selected for new files. And xcache does pass the appropriate > oss.asize opaque parameter to open. > > https://xrootd.slac.stanford.edu/doc/dev56/ofs_config.htm#_Toc116508676 > > Matevz > >> Andy >> >> >> On Thu, 16 Feb 2023, Nikolai Hartmann wrote: >> >>> Hi Andy, >>> >>> The behavior seems to be that it purges all the disks except one. After the >>> other disks now again surpassed the threshold of 95% it seemed to trigger the >>> cleanup and now i have this: >>> >>> Filesystem Type Size Used Avail Use% Mounted on >>> /dev/sdb btrfs 5,5T 5,3T 215G 97% /srv/xcache/b >>> /dev/sda btrfs 5,5T 5,0T 560G 90% /srv/xcache/a >>> /dev/sdh btrfs 5,5T 4,9T 588G 90% /srv/xcache/h >>> /dev/sdj btrfs 5,5T 4,9T 584G 90% /srv/xcache/j >>> /dev/sdf btrfs 5,5T 4,9T 580G 90% /srv/xcache/f >>> /dev/sdm btrfs 5,5T 5,0T 535G 91% /srv/xcache/m >>> /dev/sdc btrfs 5,5T 5,0T 553G 91% /srv/xcache/c >>> /dev/sdg btrfs 5,5T 4,9T 612G 90% /srv/xcache/g >>> /dev/sdi btrfs 5,5T 4,9T 596G 90% /srv/xcache/i >>> /dev/sdl btrfs 5,5T 5,0T 518G 91% /srv/xcache/l >>> /dev/sdn btrfs 5,5T 4,9T 570G 90% /srv/xcache/n >>> /dev/sde btrfs 5,5T 4,9T 593G 90% /srv/xcache/e >>> /dev/sdk btrfs 5,5T 4,8T 677G 88% /srv/xcache/k >>> /dev/sdd btrfs 5,5T 4,9T 602G 90% /srv/xcache/d >>> >>> Cheers, >>> Nikolai >>> >>> On 2/14/23 21:52, Andrew Hanushevsky wrote: >>>> Hi Matevz & Nikolai, >>>> >>>> The allocation should favor the disk with the most free space unless it's >>>> atered using the oss.alloc directive: >>>> https://urldefense.com/v3/__https://xrootd.slac.stanford.edu/doc/dev54/ofs_config.htm*_Toc89982400__;Iw!!Mih3wA!AsisYxoXis_6IdoiqK-BwdMsHfHTB41Z4-GEjaMqvO0PQHh6TqU8Sn79JUgDeJDLCvO63yQiG63Zu6syVA$ >>>> I don't think Nikolai specifies that and I don't think the pfc alters it in >>>> any way. So, I can't explain why we see that difference other than via an >>>> uneven purge. >>>> >>>> Andy >>>> >>>> >>>> On Tue, 14 Feb 2023, Matevz Tadel wrote: >>>> >>>>> Hi Nikolai, Andy, >>>>> >>>>> I saw this a long time back, 2++ years. The thing is that xcache does oss df on >>>>> the whole space and then deletes files without any knowledge of the usage on >>>>> individual disks themselves. Placement of new files should prefer the more >>>>> empty >>>>> disks though, iirc. >>>>> >>>>> I remember asking Andy about how xcache could be made aware of individual disks >>>>> and he prepared something for me but it got really complicated when I was >>>>> trying >>>>> to include this into the cache purge algorithm so I think I dropped this. >>>>> >>>>> Andy, could we sneak some debug printouts into oss new file disk selection to >>>>> see if something is going wrong there? >>>>> >>>>> Nikolai, how fast does this happen? Is it a matter of days, ie, over many purge >>>>> cycles? Is it always the same disk? >>>>> >>>>> Cheers, >>>>> Matevz >>>>> >>>>> On 2/13/23 23:21, Nikolai Hartmann wrote: >>>>>> Hi Andy, >>>>>> >>>>>> The config is the following: >>>>>> >>>>>> https://urldefense.com/v3/__https://gitlab.physik.uni-muenchen.de/etp-computing/xcache-nspawn-lrz/-/blob/086e5ade5d27fc7d5ef59448c955523e453c091f/etc/xrootd/xcache.cfg__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVipwRIRYyR$ >>>>>> The directories for `oss.localroot` and `oss.space meta` are on the system >>>>>> disk. >>>>>> The `/srv/xcache/[a-m]` are individually mounted devices. >>>>>> >>>>>> Best, >>>>>> Nikolai >>>>>> >>>>>> On 2/14/23 00:34, Andrew Hanushevsky wrote: >>>>>>> Hi Nikolai, >>>>>>> >>>>>>> Hmmm, no it seems you are the first one. Then again, not many people have a >>>>>>> multi-disk setup. So, could you send a link to your config file? It might be >>>>>>> the case that all of the metadata files wind up on the same disk and that is >>>>>>> the source of the issue here. >>>>>>> >>>>>>> Andy >>>>>>> >>>>>>> On Mon, 13 Feb 2023, Nikolai Hartmann wrote: >>>>>>> >>>>>>>> Dear xrootd-l, >>>>>>>> >>>>>>>> I'm seeing the issue that one of the disks on one of our xcache servers >>>>>>>> fills >>>>>>>> up disproportionally - that means it runs completely full until i get "no >>>>>>>> space left on device" errors without xcache running cleanup, while the other >>>>>>>> disks still have plenty of space left. My current df output: >>>>>>>> >>>>>>>> /dev/sdb btrfs 5,5T 5,2T 273G 96% /srv/xcache/b >>>>>>>> /dev/sda btrfs 5,5T 4,9T 584G 90% /srv/xcache/a >>>>>>>> /dev/sdh btrfs 5,5T 5,0T 562G 90% /srv/xcache/h >>>>>>>> /dev/sdj btrfs 5,5T 5,0T 551G 91% /srv/xcache/j >>>>>>>> /dev/sdf btrfs 5,5T 4,9T 579G 90% /srv/xcache/f >>>>>>>> [...] >>>>>>>> >>>>>>>> If you look at the first line you see that disk is 96% full while the others >>>>>>>> are around 90%. The issue occurred the first time after i built a new >>>>>>>> container for running xrootd. That change involved switching the container >>>>>>>> from centos7 to almalinux8 and changing the xrootd user id (ran chown and >>>>>>>> chgrp afterwards on the cache directories which are bind mounted). The >>>>>>>> xrootd >>>>>>>> version stayed the same (5.4.2). The high/low watermark configuration is the >>>>>>>> following: >>>>>>>> >>>>>>>> pfc.diskusage 0.90 0.95 >>>>>>>> >>>>>>>> I already tried clearing the misbehaving disk (after it ran full to 100%), >>>>>>>> but now the issue is reappearing. Has anyone seen similar issues or does it >>>>>>>> ring any bells for you? >>>>>>>> >>>>>>>> One thing i checked is the size that xrootd reports in the log for the total >>>>>>>> storage and that at least matches what i get when i sum the entries from >>>>>>>> `df`. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Nikolai >>>>>>>> >>>>>>>> ######################################################################## >>>>>>>> Use REPLY-ALL to reply to list >>>>>>>> >>>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>>> https://urldefense.com/v3/__https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1__;!!Mih3wA!DfZZQn5-SZKaGYvPW97K8SD5gDYYTy0wuUgMgQCUMhwQehl01yhKQdErjCRUz3BoZYL_nKVip_SnON6x$ >>>>> >>> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1