Hello Jerome
Below are some answers.
On Thu, 14 Jul 2005, Jerome LAURET wrote:
>
> Hello Wilko (possibly Andy),
>
> Thanks already for your many answers which clarifies
> things quite a lot but we are not entirely sure of some details.
> What we are trying to do is the following (re-explained from
> the start, it may not be possible but ..)
>
> - first, we already have files located as (no typo, note
> the path "massage" here)
>
> /data?/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
>
> where ? stands for any number between 0 and 3. Each file
> correspond to its initial source (and continuing with our
> example)
>
> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>
> so, intrinsically, /data*/starlib/reco/... is equivalent to
> /home/starreco/reco/... (different PFN, same LFN and in our
> scheme, the path is slightly modified to fall back on our
> feet PFN -> LFN)
>
>
> - Our application takes advantage of already catalogued
> PFN. We would like to do a smooth transition as there is
> about 50+ TB of data already placed ... We nicely access
> the files using rootd referencing the files as
> root://node//data1/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
> as the files are a-piori known
>
>
> - Xrootd saves us a lot of course, for example, being able
> to reduce the name space to ONLY
> root://redirector//home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>
> as if we are accessing HPSS files while Xrootd handles it
> would be reat. BUT we also would like to be able to access
> the files the usual-current way i.e. within a syntax
> indicating //node/data1.
>
> - For this to happen, Xrootd would need to understand BOTH
> /home/starreco/reco using /data* as cache (for example i.e. MPS)
> AND also be able to access /data*/starlib/reco in direct
> access mode
I think that should be possible. The config file that Pavel sent was
already set up for it.
The path have to be set as:
xrootd.export /data0
xrootd.export /data1
xrootd.export /data2
xrootd.export /data3
xrootd.export /home
olb.path r /data0
olb.path r /data1
olb.path r /data2
olb.path r /data3
olb.path r /home
oss.path /data0
oss.path /data1
oss.path /data2
oss.path /data3
oss.path /home
oss.cache /data*
I don't thing that the oss.cache directive works. It seems to me that there
is a conflict in names (I am testing this right now), maybe Andy can
answer this question.
(You might be able to use different cache names, e.g /cache? , which
are links to the /data? disks).
This setup would mean that a file will be put on disk twice if it is
requested with or without the /data prefix. Of course you could play
around with the stagecmd and when you put the file in the cache you could
create two links one in /home/... and the other one in /data?/...
But then you have the problem the the mps tools to manage a cache system
won't work and it makes the management more complicated.
You also have to modify the stagecmd and mssgwcmd to use the correct
HPSS name.
>
> If so, several questions comes to mind
>
> /home/starreco would be populated with soft-links to files named
> /data?/%home%starreco%... which shares the space with the files
> already there.
>
> * Will Xrootd be happy with that ??? [a priori, why not]
> Seems like a possible space management nightmare to me
> (shared space)
Yes, xrootd just follows the link.
>
> * What happen if in the /home/starreco/reco tree (local not
> HPSS) a soft-link is accidentely removed ?? Is it re-created ??
AFAIK, the links are not recreated. Xrootd would assume that the
file is not on disk and re-stage it.
The mps tool mps_Xeq can be used to recreate links, checking a
cache system and name space (/home/starreco), removing files, ...
> * I assume the %XX%YY% files are managed i.e. deleted along
> the line of a policy. But folowing the above question, what
> happens if a file has disappeared but tyhe soft-link remains ??
> Is Xrootd smart enough to find it and re-stage ??
At Slac we run a so called mirgr/Purge daemon that removes files
depending of disk usage and access time of files.
If the cache file is gone xrootd will try to re-stage the file
(AFAIK).
>
> * Are a lot of flat files in an identical /data?/ (i.e. files
> of the form %XX%YY% into a single dir) is read efficient ??
> Imagine that we get several 10k files in a single directory,
> will that cause a performance impact of some sort (directory
> lookup or otherwise) ??
I don't really know. For our analysis xrootd system we have only
about 1000 files per cache system. But for out production xrootd
cluster there are up to 100K files per cache. We don't see
any obvious performance problems. It might also depend
on the file system. We are using sun disks with ufs file system.
> * By the way, does "readonly" have the meaning of 'cannot open
> a new file' or really readonly (which terminology seems to
> be incompatible with writting files into /data*). I assume
> the former.
You are right. A client can not write or modify a file.
Xrootd, of course, can transfer data from MSS to disk.
>
>
> * Now for a controversial question. stagecmd command accepts
> in one of its form two arguments:
> HPSSPathAsInput localPath
> If localPath was only a "suggested path" and stagecmd would
> RETURN to Xrootd (as sstring) its final file placement,
> we could implement our own space management mechanism and
> policy. I can understand this not an item for the wish list
> but this WOULD have resolved our funny disk mapping and
> namespace issue [what it would not do is name space reduction
> at the end to a single namespace but that is another story].
> Comments ??
I guess that is really a question for Andy as it is not implemented
in xrootd and I have no idea what the cost and implication of this
feature would be.
Personally I would prefer to keep the configuration options as
small as possible.
>
> - And Pavel would like to know if someone has an example of the
> /opt/xrootd/utils/xfrcmd script (to be sent to list or privately).
>
RAL is using mps and they wrote their own xfrcmd script. They are not
using HPSS. I could ask them for the script.
Cheers,
wilko
>
> Thanks & cheers,
>
>
> Wilko Kroeger wrote:
> > Hello Pavel
> >
> > Let me try to answer your questions.
> >
> > 1) The stage command 'stagecmd':
> >
> > If a file is not on disk xrootd tries to stage the file onto disk. It
> > calls the stagecmd with two arguments:
> > stagecmd <remoteFileName> <localFileName>
> > where remoteFileName is the name in HPSS and localFileName is the one on
> > disk. Xrootd forms these names from the file name provide by the client
> > and the two prefixes oss.remoteroot and oss.localroot
> > For example:
> > the xrootd config file contains
> > oss.remoteroot /fr
> > oss.localroot /fl
> > a client requests a file like:
> > xrdcp root://dataserver//my/test.file test.file
> > in this case the stage command is called:
> > stagecmd /fr/my/test.file /fl/my/test.file
> >
> > If oss.remoteroot and oss.localroot are not specified the arguments to the
> > stagecmd is just the file name specified by the client.
> >
> > As you can see the files will always be staged into the same disk if you
> > use the oss.localroot. If you have more then one disk on an xrootd server
> > you want to use the cache system and a stagecmd that is aware of the cache
> > system.
> >
> > The xrootd actually comes with a cache aware stage command. You can find
> > it in the utils directory of an xrootd release it is called mps_Stage.
> > I haven't used it myself but I will find out how to use it.
> > The utils dir contains a few mps_XXX utils that are used to manage a cache
> > system. On the xrootd web site there is a document that describes the mps
> > system: http://xrootd.slac.stanford.edu/doc/mps_config/mps_config.htm
> >
> >
> > 2) The cache file system:
> >
> > A file that is staged into a cache file system is physically put in any of
> > the specified caches and a link between this files and the proper file
> > name is created. For example:
> >
> > Lets assume you have the following file:
> > /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> > and it is in the cache /data3:
> >
> >
> >>ls -l /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> >
> > would show:
> > /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root ->
> > /data3/%home%starreco%reco%productionHigh%FullField%P04ik%2004%f1.root
> >
> > As I mentioned above, if you want to use a cache system the stagecmd
> > has to be aware of it.
> >
> > 3) Your setup:
> >
> > In the configuration that you described below you export all the cache
> > system (/dataN) due to the olbd.path and thats maybe something you don't
> > want to do. This also means the client has to access files with the
> > name /data3/home/... making your system setup visible to the client.
> >
> > Instead, it seems to me you that would like to make files with the name
> > '/home/....' accessible to users but on the xrootd server these files are
> > stored in /dataN/... .
> >
> > You could configure your system with:
> >
> > oss.path /home dread nomig stage
> >
> > oss.cache /data*
> >
> > xrootd.export /home
> >
> > You also have to specify the stagecmd and the mssgwd command. The stage
> > command you obviously need in order to get a file out of HPSS, and the
> > mssgwd command is needed because xrootd first checks if a file is present
> > in HPSS. If you don't want xrootd to do this you could provide a
> > implementation that returns dummy data.
> > I could provide you with a little dummy script that implements some of the
> > mssgwd commands.
> >
> >
> > I don't now why all your path are migratable (mig). I suspect that one of
> > the options forces to turn on mig but I don't know which one. I have to
> > look into this.
> >
> >
> > The
> > oss.path / .....
> > is always there by default.
> >
> > I hope I clarified some of your questions. Let me know if not.
> >
> > Cheers,
> > Wilko
> >
> >
> >
> > On Wed, 13 Jul 2005, Pavel Jakl wrote:
> >
> >
> >>Hi Wilko,
> >>
> >>many thanks for advice. With directive oss.check client is redirected to
> >>other node and staging of requested file is started.
> >>I have some other questions, if you can help me, it will be super.
> >>
> >>I want to stage a file from HPSS for example with path like:
> >>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> >>5040131_raw_2010003.MuDst.root
> >>
> >>and copy to path /data0 or /data1 or /data2 or /data3 which I've
> >>specified with directives:
> >>
> >>oss.path /data0 dread nomig stage
> >>oss.path /data1 dread nomig stage
> >>oss.path /data2 dread nomig stage
> >>oss.path /data3 dread nomig stage
> >>
> >>My question is if my thoughts are right and file with HPPS path:
> >>
> >>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> >>5040131_raw_2010003.MuDst.root
> >>
> >>will be staged to local path:
> >>
> >>/data0/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_ph
> >>ysics_5040131_raw_2010003.MuDst.root
> >>i.e that /data0 is path to ve pre-pended to local path. Or is there some
> >>other directive, which can help me to do that ?
> >>
> >>My second question is if I need to have specified oss.mssgwcmd command
> >>when I don't plan to migrate any files from local disk to HPSS ?
> >>
> >>The third question is about oss.cache directive. Can I have the location
> >>of cache in the same directory as I have the files to export ?
> >>For example, I have all files in directories:
> >>/data0, /data1, /data2 or /data3 (i export with directive xrootd.export)
> >>and can I have a oss.cache /data*.
> >>
> >>My fourth problem is with oss.path again, I've specified this:
> >>oss.path /data0 dread nomig stage
> >>oss.path /data1 dread nomig stage
> >>oss.path /data2 dread nomig stage
> >>oss.path /data3 dread nomig stage
> >>
> >>but in log file is this:
> >>oss.path /data3 r/o check dread mig nomkeep nomlock nommap norcreate
> >>stage
> >>oss.path /data2 r/o check dread mig nomkeep nomlock nommap norcreate
> >>stage
> >>oss.path /data1 r/o check dread mig nomkeep nomlock nommap norcreate
> >>stage
> >>oss.path /data0 r/o check dread mig nomkeep nomlock nommap norcreate
> >>stage
> >>
> >>as you can see with mig directive and in addation is here a line:
> >>
> >>oss.path / r/o check dread mig nomkeep nomlock nommap norcreate stage
> >>
> >>Is this correct ? Becaouse I dont want to stage any files to path /. (Or
> >>is needed due to that / is parent directory ?)
> >>
> >>Thank you
> >>Pavel
> >>
> >>On Tue, 12 Jul 2005 20:46:20 -0700 (PDT)
> >>Wilko Kroeger <[log in to unmask]> wrote:
> >>
> >>
> >>>Hello Pavel
> >>>
> >>>I suspect that this is a problem with the mss configuration. In the
> >>>dev. version the mss command is ignored if the path is only stage-able
> >>>and nodread, nocheck, nomig are specified.
> >>>Do you see in the xrdlog file the line
> >>> mssgwcmd ignored; no MSS paths present.
> >>>from the xrootd startup ?
> >>>
> >>>If this is the case you could add
> >>>oss.check
> >>>to your config which should cause xrootd to use the mssgwcmd command.
> >>>(oss.dread ot oss.rcreate should also do the trick). But if you
> >>>specify any of these options xrootd will behave a little bit different
> >>>from how it is configured now.
> >>>
> >>>The problem has been fixed in CVS and will be in the next release.
> >>>
> >>>
> >>>Cheers,
> >>> Wilko
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>On Tue, 12 Jul 2005, Pavel Jakl wrote:
> >>>
> >>>
> >>>>Hi for some time,
> >>>>
> >>>>Firstly, we've installed the latest development version of xrootd on
> >>>>100 nodes (1 redirector + 1 supervisor + 98 dataservers) with the
> >>>>support of open load balancing and everything is working great and
> >>>>super. I would like to say, brilliant work.
> >>>>
> >>>>To my problem, I want to enable the MSS interface. We've implemented
> >>>>scripts for performing meta-data operations and staging files from
> >>>>HPSS. (directives oss.mssgwcmd, oss.stagecmd). I 've inserted
> >>>>everything in config file for dataserver and tried to obtain a file
> >>>>from HPSS. In redirector log I found this error message:
> >>>>
> >>>> 050712 19:35:48 18366 odc_Locate: starlib.10447:13@rcas6230 given
> >>>> error msg 'No servers are available to read the file.' by xrdstar
> >>>> path=/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/s
> >>>> t_physics_5040131_raw_2010003.MuDst.root
> >>>>
> >>>>and found out that scripts was not even called once time. (this I
> >>>>know from debug support of my script). My question is, if I have
> >>>>something wrong in my configuration file or I forgot something to
> >>>>add or I skiped something in reading of documentation.
> >>>>
> >>>>I am enclosing a configuration file for dataserver.
> >>>>
> >>>>Thank you for any advice
> >>>>Pavel
> >>>>
> >>>>
> >>>>
> >>
>
> --
> ,,,,,
> ( o o )
> --m---U---m--
> Jerome
>
