On 2019-11-14 12:39, Sam Skipsey wrote:
> On Thu, 14 Nov 2019 at 20:27, Matevž Tadel <[log in to unmask]> wrote:
>>
>> Hi,
>>
>> You are missing oss.localroot ... this has to be always defined. That's
>> where symlinks to various oss.space instances go.
>>
>
> ...how does this work (and why isn't it described in any detail how
> this applies to oss.spaces in the documentation?)
I think Andy should explain this (and fix the docs) ... as it might be
my mental model is incomplete.
>> Looking at your config, you don't seem to need oss.space and pfc.spaces
>> at all, you just need oss.localroot /storage0/
>>
>
> No, we have /storage0 through /storage4.
>
>> It seems the disks are aggregated in some other ways, not by being added
>> to any space.
>>
>
> That's... partly true. But the /storage* filesystem locations are all
> distinct, and associated with different devices.
>
>> public is the default space used by xcache, for both data and metadata
>> (so what you specify are actually the defaults).
>>
>
> I know, but since it wasn't working, I thought I'd put it in
> explicitly just in case.
>
>> xcache allows you to separate spaces for data and metadata, often it is
>> useful to have metadata and localroot on ssd, or at least on a separate
>> disk / controller.
>>
>
> Indeed... as noted, the problem is that there's no documentation about
> how to associate a space with the localroot, as the documentation
> doesn't talk about spaces in this context.
All spaces are associate with the localroot implicitly, I believe ...
localroot just holds symlinks to instaniated files in various spaces.
You can think of localroot as the namespace representation ... with
actual data being stored on other spaces.
Now, the question that arises for me is, how you associate parts of
namespace (export) with an oss.space -- and I believe this is also what
confuses you.
In xcache we do it explicitly by writing data files to the "data" space
and cinfo files to the "meta" space. This way they can land on different
storage types and can be separated as cinfo files get access often
during purge cycle.
>> See the relevant lines from a config we use at UCSD below.
>>
>> It turns out that for cache it is better to use raw disks than raid5/6
>> or zfs as you get independent streams for each file being opened going
>> to specific disks, both on input and output. Data loss due to a lost
>> disk is not a problem for caches.
>>
>> I hope this is clearer now :)
>>
>
> Not really: I still need to know how you associate the localroot with a space.
See above + I hope Andy can further clarify it for you.
>> Matevz
>>
>> oss.localroot /xcache-root # On SSD
>>
>> oss.space data /data1/xcache # Spinning disks (we use xfs)
>> oss.space data /data2/xcache
>> oss.space data /data3/xcache
>> oss.space data /data4/xcache
>> oss.space data /data5/xcache
>> oss.space data /data6/xcache
>> oss.space data /data7/xcache
>> oss.space data /data8/xcache
>> oss.space data /data9/xcache
>> oss.space data /data10/xcache
>> oss.space data /data11/xcache
>> oss.space data /data12/xcache
>>
>> oss.space meta /xcache-meta # On SSD
>>
>> # Use space "data" for data, "meta" for meta-data/cinfo files
>> pfc.spaces data meta
>
> Yes. So... is /xcache-root a real path? Is /xcache-meta a real path?
Yes, both are real paths you can write to :)
> This doesn't help me with what I should point oss.localroot at.
To a directory, preferably on a SSD. And the same for oss.space meta.
localroot will only hold sym-links so it needs not be very large. meta
will only hold cinfo files so it will be something like 1 kB per file +
1-bit per-block of data files, also rather smallish.
E.g., for blocksize of 512 kB:
[1311] root@xcache-00 /# du -sch xcache-root xcache-meta /data{1..12}
3.8M xcache-root
31M xcache-meta
1.8T /data1
1.8T /data2
1.8T /data3
1.8T /data4
1.8T /data5
1.8T /data6
1.8T /data7
1.8T /data8
1.8T /data9
1.8T /data10
1.8T /data11
1.8T /data12
22T total
Matevz
> Sam
>
>>
>> On 2019-11-14 11:50, Sam Skipsey wrote:
>>> Hello,
>>> So, I've tried various interpretations of the documentation, but I'm
>>> obviously missing something.
>>>
>>> I'm trying to configure an Xrootd Proxy Cache, backed by storage
>>> aggregated into an oss.space.
>>>
>>> The cache is configured like:
>>>
>>>
>>> oss.space public /storage0*
>>> #oss.space public default /
>>>
>>> ofs.osslib libXrdPss.so
>>> pss.cachelib libXrdFileCache.so
>>> pfc.ram 16g
>>> all.trace all
>>> all.log all
>>> pfs.diskusage 0.90 0.95
>>>
>>> all.export /xroot:/ outplace
>>> all.export /root:/ outplace
>>>
>>> all.export / outplace
>>>
>>> pss.origin svr01.beowulf.cluster:1094
>>> xrd.allow host *.beowulf.cluster
>>>
>>> pfc.blocksize 256k
>>>
>>> pfc.spaces public public
>>>
>>>
>>>
>>> where the /storage0* spaces are all writable and owned by the xrootd
>>> user (and when we start up the server, it creates "public" directories
>>> in each of them).
>>>
>>> However, whenever I try to access a file through the proxy, it logs
>>> that it couldn't write the file to the disk (specifying its path as if
>>> it was going to write it to the storage with no prefix - that is,
>>> outside of the specified "public" space, in the literal root
>>> filesystem).
>>> If this was a non-spaces based system, I'd specify oss.localroot, but
>>> that doesn't seem to apply to a space?
>>> I tried adding the outplace directives to the all.exports, but this
>>> hasn't changed anything.
>>>
>>> What am I missing? None of the example configurations I can see with
>>> oss.spaces seem to need any additional configuration directives...
>>>
>>> Sam
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
