Ah, okay, so when you're using spaces, the localroot is a "fake" (but
real filesystem directory) which just symlinks into the spaces, as a
sort of "lookup table" for which space to actually find a given file?
That's interesting. (And this is definitely not explained *at all* in
the documentation, so that needs fixed. The oss.spaces stuff just says
that it is a means of aggregating storage spaces and doesn't say that
you need a localroot too, and the oss.localroot stuff doesn't mention
how it relates to spaces at all.)
On Thu, 14 Nov 2019 at 21:13, Matevž Tadel <[log in to unmask]> wrote:
>
>
>
> On 2019-11-14 12:39, Sam Skipsey wrote:
> > On Thu, 14 Nov 2019 at 20:27, Matevž Tadel <[log in to unmask]> wrote:
> >>
> >> Hi,
> >>
> >> You are missing oss.localroot ... this has to be always defined. That's
> >> where symlinks to various oss.space instances go.
> >>
> >
> > ...how does this work (and why isn't it described in any detail how
> > this applies to oss.spaces in the documentation?)
>
> I think Andy should explain this (and fix the docs) ... as it might be
> my mental model is incomplete.
>
> >> Looking at your config, you don't seem to need oss.space and pfc.spaces
> >> at all, you just need oss.localroot /storage0/
> >>
> >
> > No, we have /storage0 through /storage4.
> >
> >> It seems the disks are aggregated in some other ways, not by being added
> >> to any space.
> >>
> >
> > That's... partly true. But the /storage* filesystem locations are all
> > distinct, and associated with different devices.
> >
> >> public is the default space used by xcache, for both data and metadata
> >> (so what you specify are actually the defaults).
> >>
> >
> > I know, but since it wasn't working, I thought I'd put it in
> > explicitly just in case.
> >
> >> xcache allows you to separate spaces for data and metadata, often it is
> >> useful to have metadata and localroot on ssd, or at least on a separate
> >> disk / controller.
> >>
> >
> > Indeed... as noted, the problem is that there's no documentation about
> > how to associate a space with the localroot, as the documentation
> > doesn't talk about spaces in this context.
>
> All spaces are associate with the localroot implicitly, I believe ...
> localroot just holds symlinks to instaniated files in various spaces.
> You can think of localroot as the namespace representation ... with
> actual data being stored on other spaces.
>
> Now, the question that arises for me is, how you associate parts of
> namespace (export) with an oss.space -- and I believe this is also what
> confuses you.
>
> In xcache we do it explicitly by writing data files to the "data" space
> and cinfo files to the "meta" space. This way they can land on different
> storage types and can be separated as cinfo files get access often
> during purge cycle.
>
> >> See the relevant lines from a config we use at UCSD below.
> >>
> >> It turns out that for cache it is better to use raw disks than raid5/6
> >> or zfs as you get independent streams for each file being opened going
> >> to specific disks, both on input and output. Data loss due to a lost
> >> disk is not a problem for caches.
> >>
> >> I hope this is clearer now :)
> >>
> >
> > Not really: I still need to know how you associate the localroot with a space.
>
> See above + I hope Andy can further clarify it for you.
>
> >> Matevz
> >>
> >> oss.localroot /xcache-root # On SSD
> >>
> >> oss.space data /data1/xcache # Spinning disks (we use xfs)
> >> oss.space data /data2/xcache
> >> oss.space data /data3/xcache
> >> oss.space data /data4/xcache
> >> oss.space data /data5/xcache
> >> oss.space data /data6/xcache
> >> oss.space data /data7/xcache
> >> oss.space data /data8/xcache
> >> oss.space data /data9/xcache
> >> oss.space data /data10/xcache
> >> oss.space data /data11/xcache
> >> oss.space data /data12/xcache
> >>
> >> oss.space meta /xcache-meta # On SSD
> >>
> >> # Use space "data" for data, "meta" for meta-data/cinfo files
> >> pfc.spaces data meta
> >
> > Yes. So... is /xcache-root a real path? Is /xcache-meta a real path?
>
> Yes, both are real paths you can write to :)
>
> > This doesn't help me with what I should point oss.localroot at.
>
> To a directory, preferably on a SSD. And the same for oss.space meta.
> localroot will only hold sym-links so it needs not be very large. meta
> will only hold cinfo files so it will be something like 1 kB per file +
> 1-bit per-block of data files, also rather smallish.
>
> E.g., for blocksize of 512 kB:
>
> [1311] root@xcache-00 /# du -sch xcache-root xcache-meta /data{1..12}
> 3.8M xcache-root
> 31M xcache-meta
> 1.8T /data1
> 1.8T /data2
> 1.8T /data3
> 1.8T /data4
> 1.8T /data5
> 1.8T /data6
> 1.8T /data7
> 1.8T /data8
> 1.8T /data9
> 1.8T /data10
> 1.8T /data11
> 1.8T /data12
> 22T total
>
> Matevz
>
> > Sam
> >
> >>
> >> On 2019-11-14 11:50, Sam Skipsey wrote:
> >>> Hello,
> >>> So, I've tried various interpretations of the documentation, but I'm
> >>> obviously missing something.
> >>>
> >>> I'm trying to configure an Xrootd Proxy Cache, backed by storage
> >>> aggregated into an oss.space.
> >>>
> >>> The cache is configured like:
> >>>
> >>>
> >>> oss.space public /storage0*
> >>> #oss.space public default /
> >>>
> >>> ofs.osslib libXrdPss.so
> >>> pss.cachelib libXrdFileCache.so
> >>> pfc.ram 16g
> >>> all.trace all
> >>> all.log all
> >>> pfs.diskusage 0.90 0.95
> >>>
> >>> all.export /xroot:/ outplace
> >>> all.export /root:/ outplace
> >>>
> >>> all.export / outplace
> >>>
> >>> pss.origin svr01.beowulf.cluster:1094
> >>> xrd.allow host *.beowulf.cluster
> >>>
> >>> pfc.blocksize 256k
> >>>
> >>> pfc.spaces public public
> >>>
> >>>
> >>>
> >>> where the /storage0* spaces are all writable and owned by the xrootd
> >>> user (and when we start up the server, it creates "public" directories
> >>> in each of them).
> >>>
> >>> However, whenever I try to access a file through the proxy, it logs
> >>> that it couldn't write the file to the disk (specifying its path as if
> >>> it was going to write it to the storage with no prefix - that is,
> >>> outside of the specified "public" space, in the literal root
> >>> filesystem).
> >>> If this was a non-spaces based system, I'd specify oss.localroot, but
> >>> that doesn't seem to apply to a space?
> >>> I tried adding the outplace directives to the all.exports, but this
> >>> hasn't changed anything.
> >>>
> >>> What am I missing? None of the example configurations I can see with
> >>> oss.spaces seem to need any additional configuration directives...
> >>>
> >>> Sam
> >>>
> >>> ########################################################################
> >>> Use REPLY-ALL to reply to list
> >>>
> >>> To unsubscribe from the XROOTD-L list, click the following link:
> >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> >>>
> >>
> >> ########################################################################
> >> Use REPLY-ALL to reply to list
> >>
> >> To unsubscribe from the XROOTD-L list, click the following link:
> >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
