Hi Andy,
On 05/16/18 15:13, Andrew Hanushevsky wrote:
> Well, that sounds like a good step forward. If it's simple to do then I'd say go
> ahead. But I think it only forestalls people clamoring for a plugin (I've been
> down that road before and no matter how great one idea is someone else thinks
> they have a better idea :-).
Yes, it's really easy (well) to implement on top of what we have. I agree we'll
need a purge plugin eventually. But I also think we should have a reasonable
default implementation so let's push this cart for a bit longer :)
> P.S. You may want to replace share count by a simple percentage (or have a way
> of specifying it along withshare count -- though it's likely to be too
> confusing). The share count idea assumes all parties are equal. So, a share
> count of 3 gives everyone a 33% share. But that's not what I might want. I may
> want one party to get 50% of the space and the other two 25% (I might even want
> to over commit for my own personal bizarre reasons). So, consider that.
Yes, this is all covered. Share-count is just for when disks are full. This
scheme also supports over-committing (by virtue of file min / nominal / max
settings).
So I'll go ahead and implement this.
Matevz
> Andy
>
> On Wed, 16 May 2018, Matevz Tadel wrote:
>
>> On 05/16/18 13:44, Andrew Hanushevsky wrote:
>>> Well, there is a full on hammer solution by just using a specialized oss
>>> plugin. The cache gets it's usage values from this plugin. You could wrap it
>>> and override the space usage method. Of course, like I said this is a rather
>>> heavy duy fix. There is merit to Wei's solution by having a lihhtweight
>>> plugin that either returns space usage and is used instead of the oss plugin
>>> if it was specified. Alternatively, have some kind of decision plugin that us
>>> used to determine if you have enough space left. I think a plugin approach
>>> will be the only way we satisfy everyone's conflicting ideas of what "space"
>>> really means.
>>
>> I have a proposal. Supersede pfc.diskusage with
>>
>> pfc.storagelimits files <low> <nominal> <high> disk <low> <high>
>> share-count <num> purge-interval <time>
>>
>> This would:
>> a) keep usage of actual cached files between F_nominal and F_high;
>>
>> b) if total disk usage gets above D_high, cache will retreat to F_low (using
>> share-count to determine what fraction of required space to concede on each
>> purge cycle.
>>
>> c) when cache would have to purge below F_low, it stops caching new files, it
>> acts as ram-only proxy for new files until disk becomes available. It will not
>> purge beyond F_low.
>>
>> Additionally, we can add:
>>
>> pfc.purge older-than <time> not-accessed-in <time>
>>
>> What do you think?
>>
>> Matevz
>>
>>> Andy
>>>
>>> On Wed, 16 May 2018, Matevz Tadel wrote:
>>>
>>>> On 05/15/18 20:34, Yang, Wei wrote:
>>>>> I am evaluating Xcache at NERSC, using their shared filesystems. I run into
>>>>> two issues
>>>>>
>>>>> 1. "pfc.diskusage low high" . I want to set 5TB and 8TB respectively. But
>>>>> the diskusage is based on filesystem's total and used space. The total and
>>>>> used of the whole filesystem is 7PB and 5.8PB. So Xcache always thinks that
>>>>> I run out of space and remove everything. Anyway to change this?
>>>>
>>>> We would need to add a flag to the pfc.diskusage ... like total (what we
>>>> have now) vs. actual (what you are asking for now). Hmmh, or would one want
>>>> to have both controls at the same time, i.e., to start purging if either of
>>>> the following is true:
>>>> a) cache is using more than X GB (or X % of total space on the partition);
>>>> b) the disk partition has more than X GB used (or is more than X % full).
>>>>
>>>> Let me think ... I think it might make sense to have both options, then
>>>> several xcache instances can use the same pool of disks without stepping on
>>>> each other's toes.
>>>>
>>>>> Is it possible to allow a call a simple script to determine the current
>>>>> used space? Also set the interval of cleaning?
>>>>
>>>> No, we use OfsDF (or something like that). But we need to loop over all
>>>> cinfo files to decide which to remove so we can also calculate actual disk
>>>> usage while we're doing this.
>>>>
>>>> Andy, a question. On most linux FSs, when we have a sparse file, only blocks
>>>> that are actually written "consume" disk space. Is this universally true? Is
>>>> it ok if I only use stored block-count as the size estimate for actual disk
>>>> usage? Can we know the block size of the FS? The problem, I think, is only
>>>> the last, not fully occupied block .. and depending on the cache block size
>>>> and the number of files in the cache this could lead to a relatively large
>>>> "rounding error".
>>>>
>>>> Time interval can be changed ... oh ... it's not in THE DOCS :)
>>>>
>>>> pfc.diskusage fracLow fracHigh
>>>>
>>>> is actually
>>>>
>>>> pfc.diskusage fracLow fracHigh [sleep time]
>>>>
>>>> where time is interval between purges checks, clamped between 60 and 3600
>>>> seconds.
>>>>
>>>> https://github.com/xrootd/xrootd/blob/master/src/XrdFileCache/XrdFileCacheConfiguration.cc#L337
>>>>
>>>> I have no clue about 2 below ...
>>>>
>>>> Matevz
>>>>
>>>>> 2. Putting the cache space on NERSC GPFS works but putting it on NERSC
>>>>> Luster (not sure what version) does not. When the cache is on Luster,
>>>>> nothing is cached and the log shows this:
>>>>>
>>>>> 180515 14:49:30 2801 XrootdXeq: yangw.3336:27@dtn04 pub IPv4 login
>>>>> 180515 14:49:30 2801 XrdFileCache_Manager: info Cache::Attach()
>>>>> root://27@localfile:1094//atlas/rucio/transient/31/c5/panda.HITS.14104313._000001.pool.root.1.14104313-3930415979-13437313503-47-10.zip
>>>>> 180515 14:49:30 2801 ofs_FAttr: Unable to set attr XrdFrm.Pfn from
>>>>> /global/cscratch1/sd/yangw/xcache/dtn04.nersc.gov/xrd/datafiles/data/01/4D55FB5AE70A000000008037cd1501000000404%;
>>>>> operation not supported
>>>>> 180515 14:49:30 2801 XrdFileCache_File: error File::Open() Create failed
>>>>> for data file
>>>>> /atlas/rucio/transient/31/c5/panda.HITS.14104313._000001.pool.root.1.14104313-3930415979-13437313503-47-10.zip,
>>>>> err=Operation not supported
>>>>> /atlas/rucio/transient/31/c5/panda.HITS.14104313._000001.pool.root.1.14104313-3930415979-13437313503-47-10.zip
>>>>> 180515 14:49:42 2801 XrdFileCache_IO: info IOEntireFile::Detach() 0x1cd6600
>>>>> 180515 14:49:42 2801 XrootdXeq: yangw.3336:27@dtn04 disc 0:00:12
>>>>>
>>>>> --
>>>>> Wei Yang | [log in to unmask] | 650-926-3338(O)
>>>>>
>>>>> ########################################################################
>>>>> Use REPLY-ALL to reply to list
>>>>>
>>>>> To unsubscribe from the XCACHE-L list, click the following link:
>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XCACHE-L&A=1
>>>>>
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XCACHE-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XCACHE-L&A=1
>>>>
>>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XCACHE-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XCACHE-L&A=1
|