Hello again Wilko,
thank you for your answers (some of them were very usefull for me) . After your advice I tried to set MPS beacause we need to use more than 1 cache file system.
I have some problem with directive mps.basedir, let me explain to you what I want:
I am requesting this file from HPSS:
/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/file1.root
and want to stage from HPSS to cache system. (/data0 or /data1 or /data2 ...)
Firstly MPS is trying to create directories for symlink:
/home/starreco/reco ... ()
but this a problem, because I am running xrrotd as user starlib and I don't have permission to this directory.
So I decided to add directive
mps.basedir /home/starlib
thank every file will be staged (its symlink to cache file system) like:
/home/starlib/home/starreco/reco ....
But as I mentioned above, the directive basedir is not working and still want to stage file as /home/starreco.
Can you look at my configuration file, if something is wrong ?
Many thanks
Pavel
On Fri, 15 Jul 2005 13:05:43 -0700 (PDT)
Wilko Kroeger <[log in to unmask]> wrote:
>
> Hello Pavel
>
> On Fri, 15 Jul 2005, Pavel Jakl wrote:
>
> > Hello again,
> >
> > In the MPS manual and mps_* scripts, there are things like
> >
> > $Config{xfrcmd} = ($isMPS ? '/opt/xrootd/utils/xfrcmd'
> > : '/usr/etc/ooss/pftp_client');
> >
> > and similar switch for port, nodes, account name ... while both
> > seem to accept ftp-like commands. Could you please explain what
> > was the philosophy of having separate commands for oss/mps (but
> > using the same scripts) and why different port/nodes ?? This is
> > not entirely clear to us ...
>
> The ooss scripts are used at SLAC to manage the cache files systems.
> The mps scripts evolved out of the ooss scripts and they are kept
> backwards compatible, meaning that they would work with the ooss scripts
> installed in /usr/etc/ooss. The $isMPS basically decides if the MPS
> scripts that come with xrootd are used or the ooss scripts (installed in
> /usr/etc/ooss). This allows site that have ooss installed (like SLAC) to
> use the mps scripts.
>
> >
> > Since stagecmd for oss is pftp and mps stagecmd is
> > xfrcmd, it makes us puzzled as per what xfrcmd does in addition
> > of pftp. This does not appear to be explained in the documentation.
>
> The '/opt/xrootd/utils/xfrcmd' should do the same thing that
> '/usr/etc/ooss/pftp_client' does for HPSS.
> Different sites use different MSS systems and even if they use the same
> system the access methods could be different. Therefore the mps scripts were
> written so that an xrootd site can provide their own implementation for
> mss access without modifying the mps scripts. If you are using pftp to access
> HPSS you could just call it from within /opt/xrootd/utils/xfrcmd.
>
> I hope this clarified your issues a littele bit.
>
> Cheers,
> Wilko
>
> >
> >
> > Jerome & Pavel
> >
> >
> > On Thu, 14 Jul 2005 20:50:39 -0400
> > Jerome LAURET <[log in to unmask]> wrote:
> >
> > >
> > > Hello Wilko (possibly Andy),
> > >
> > > Thanks already for your many answers which clarifies
> > > things quite a lot but we are not entirely sure of some details.
> > > What we are trying to do is the following (re-explained from
> > > the start, it may not be possible but ..)
> > >
> > > - first, we already have files located as (no typo, note
> > > the path "massage" here)
> > >
> > > /data?/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
> > >
> > > where ? stands for any number between 0 and 3. Each file
> > > correspond to its initial source (and continuing with our
> > > example)
> > >
> > > /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> > >
> > > so, intrinsically, /data*/starlib/reco/... is equivalent to
> > > /home/starreco/reco/... (different PFN, same LFN and in our
> > > scheme, the path is slightly modified to fall back on our
> > > feet PFN -> LFN)
> > >
> > >
> > > - Our application takes advantage of already catalogued
> > > PFN. We would like to do a smooth transition as there is
> > > about 50+ TB of data already placed ... We nicely access
> > > the files using rootd referencing the files as
> > > root://node//data1/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
> > > as the files are a-piori known
> > >
> > >
> > > - Xrootd saves us a lot of course, for example, being able
> > > to reduce the name space to ONLY
> > > root://redirector//home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> > >
> > > as if we are accessing HPSS files while Xrootd handles it
> > > would be reat. BUT we also would like to be able to access
> > > the files the usual-current way i.e. within a syntax
> > > indicating //node/data1.
> > >
> > > - For this to happen, Xrootd would need to understand BOTH
> > > /home/starreco/reco using /data* as cache (for example i.e. MPS)
> > > AND also be able to access /data*/starlib/reco in direct
> > > access mode
> > >
> > > If so, several questions comes to mind
> > >
> > > /home/starreco would be populated with soft-links to files named
> > > /data?/%home%starreco%... which shares the space with the files
> > > already there.
> > >
> > > * Will Xrootd be happy with that ??? [a priori, why not]
> > > Seems like a possible space management nightmare to me
> > > (shared space)
> > >
> > > * What happen if in the /home/starreco/reco tree (local not
> > > HPSS) a soft-link is accidentely removed ?? Is it re-created ??
> > >
> > > * I assume the %XX%YY% files are managed i.e. deleted along
> > > the line of a policy. But folowing the above question, what
> > > happens if a file has disappeared but tyhe soft-link remains ??
> > > Is Xrootd smart enough to find it and re-stage ??
> > >
> > > * Are a lot of flat files in an identical /data?/ (i.e. files
> > > of the form %XX%YY% into a single dir) is read efficient ??
> > > Imagine that we get several 10k files in a single directory,
> > > will that cause a performance impact of some sort (directory
> > > lookup or otherwise) ??
> > >
> > > * By the way, does "readonly" have the meaning of 'cannot open
> > > a new file' or really readonly (which terminology seems to
> > > be incompatible with writting files into /data*). I assume
> > > the former.
> > >
> > >
> > > * Now for a controversial question. stagecmd command accepts
> > > in one of its form two arguments:
> > > HPSSPathAsInput localPath
> > > If localPath was only a "suggested path" and stagecmd would
> > > RETURN to Xrootd (as sstring) its final file placement,
> > > we could implement our own space management mechanism and
> > > policy. I can understand this not an item for the wish list
> > > but this WOULD have resolved our funny disk mapping and
> > > namespace issue [what it would not do is name space reduction
> > > at the end to a single namespace but that is another story].
> > > Comments ??
> > >
> > > - And Pavel would like to know if someone has an example of the
> > > /opt/xrootd/utils/xfrcmd script (to be sent to list or privately).
> > >
> > >
> > > Thanks & cheers,
> > >
> > >
> > > Wilko Kroeger wrote:
> > > > Hello Pavel
> > > >
> > > > Let me try to answer your questions.
> > > >
> > > > 1) The stage command 'stagecmd':
> > > >
> > > > If a file is not on disk xrootd tries to stage the file onto disk. It
> > > > calls the stagecmd with two arguments:
> > > > stagecmd <remoteFileName> <localFileName>
> > > > where remoteFileName is the name in HPSS and localFileName is the one on
> > > > disk. Xrootd forms these names from the file name provide by the client
> > > > and the two prefixes oss.remoteroot and oss.localroot
> > > > For example:
> > > > the xrootd config file contains
> > > > oss.remoteroot /fr
> > > > oss.localroot /fl
> > > > a client requests a file like:
> > > > xrdcp root://dataserver//my/test.file test.file
> > > > in this case the stage command is called:
> > > > stagecmd /fr/my/test.file /fl/my/test.file
> > > >
> > > > If oss.remoteroot and oss.localroot are not specified the arguments to the
> > > > stagecmd is just the file name specified by the client.
> > > >
> > > > As you can see the files will always be staged into the same disk if you
> > > > use the oss.localroot. If you have more then one disk on an xrootd server
> > > > you want to use the cache system and a stagecmd that is aware of the cache
> > > > system.
> > > >
> > > > The xrootd actually comes with a cache aware stage command. You can find
> > > > it in the utils directory of an xrootd release it is called mps_Stage.
> > > > I haven't used it myself but I will find out how to use it.
> > > > The utils dir contains a few mps_XXX utils that are used to manage a cache
> > > > system. On the xrootd web site there is a document that describes the mps
> > > > system: http://xrootd.slac.stanford.edu/doc/mps_config/mps_config.htm
> > > >
> > > >
> > > > 2) The cache file system:
> > > >
> > > > A file that is staged into a cache file system is physically put in any of
> > > > the specified caches and a link between this files and the proper file
> > > > name is created. For example:
> > > >
> > > > Lets assume you have the following file:
> > > > /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> > > > and it is in the cache /data3:
> > > >
> > > >
> > > >>ls -l /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> > > >
> > > > would show:
> > > > /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root ->
> > > > /data3/%home%starreco%reco%productionHigh%FullField%P04ik%2004%f1.root
> > > >
> > > > As I mentioned above, if you want to use a cache system the stagecmd
> > > > has to be aware of it.
> > > >
> > > > 3) Your setup:
> > > >
> > > > In the configuration that you described below you export all the cache
> > > > system (/dataN) due to the olbd.path and thats maybe something you don't
> > > > want to do. This also means the client has to access files with the
> > > > name /data3/home/... making your system setup visible to the client.
> > > >
> > > > Instead, it seems to me you that would like to make files with the name
> > > > '/home/....' accessible to users but on the xrootd server these files are
> > > > stored in /dataN/... .
> > > >
> > > > You could configure your system with:
> > > >
> > > > oss.path /home dread nomig stage
> > > >
> > > > oss.cache /data*
> > > >
> > > > xrootd.export /home
> > > >
> > > > You also have to specify the stagecmd and the mssgwd command. The stage
> > > > command you obviously need in order to get a file out of HPSS, and the
> > > > mssgwd command is needed because xrootd first checks if a file is present
> > > > in HPSS. If you don't want xrootd to do this you could provide a
> > > > implementation that returns dummy data.
> > > > I could provide you with a little dummy script that implements some of the
> > > > mssgwd commands.
> > > >
> > > >
> > > > I don't now why all your path are migratable (mig). I suspect that one of
> > > > the options forces to turn on mig but I don't know which one. I have to
> > > > look into this.
> > > >
> > > >
> > > > The
> > > > oss.path / .....
> > > > is always there by default.
> > > >
> > > > I hope I clarified some of your questions. Let me know if not.
> > > >
> > > > Cheers,
> > > > Wilko
> > > >
> > > >
> > > >
> > > > On Wed, 13 Jul 2005, Pavel Jakl wrote:
> > > >
> > > >
> > > >>Hi Wilko,
> > > >>
> > > >>many thanks for advice. With directive oss.check client is redirected to
> > > >>other node and staging of requested file is started.
> > > >>I have some other questions, if you can help me, it will be super.
> > > >>
> > > >>I want to stage a file from HPSS for example with path like:
> > > >>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> > > >>5040131_raw_2010003.MuDst.root
> > > >>
> > > >>and copy to path /data0 or /data1 or /data2 or /data3 which I've
> > > >>specified with directives:
> > > >>
> > > >>oss.path /data0 dread nomig stage
> > > >>oss.path /data1 dread nomig stage
> > > >>oss.path /data2 dread nomig stage
> > > >>oss.path /data3 dread nomig stage
> > > >>
> > > >>My question is if my thoughts are right and file with HPPS path:
> > > >>
> > > >>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> > > >>5040131_raw_2010003.MuDst.root
> > > >>
> > > >>will be staged to local path:
> > > >>
> > > >>/data0/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_ph
> > > >>ysics_5040131_raw_2010003.MuDst.root
> > > >>i.e that /data0 is path to ve pre-pended to local path. Or is there some
> > > >>other directive, which can help me to do that ?
> > > >>
> > > >>My second question is if I need to have specified oss.mssgwcmd command
> > > >>when I don't plan to migrate any files from local disk to HPSS ?
> > > >>
> > > >>The third question is about oss.cache directive. Can I have the location
> > > >>of cache in the same directory as I have the files to export ?
> > > >>For example, I have all files in directories:
> > > >>/data0, /data1, /data2 or /data3 (i export with directive xrootd.export)
> > > >>and can I have a oss.cache /data*.
> > > >>
> > > >>My fourth problem is with oss.path again, I've specified this:
> > > >>oss.path /data0 dread nomig stage
> > > >>oss.path /data1 dread nomig stage
> > > >>oss.path /data2 dread nomig stage
> > > >>oss.path /data3 dread nomig stage
> > > >>
> > > >>but in log file is this:
> > > >>oss.path /data3 r/o check dread mig nomkeep nomlock nommap norcreate
> > > >>stage
> > > >>oss.path /data2 r/o check dread mig nomkeep nomlock nommap norcreate
> > > >>stage
> > > >>oss.path /data1 r/o check dread mig nomkeep nomlock nommap norcreate
> > > >>stage
> > > >>oss.path /data0 r/o check dread mig nomkeep nomlock nommap norcreate
> > > >>stage
> > > >>
> > > >>as you can see with mig directive and in addation is here a line:
> > > >>
> > > >>oss.path / r/o check dread mig nomkeep nomlock nommap norcreate stage
> > > >>
> > > >>Is this correct ? Becaouse I dont want to stage any files to path /. (Or
> > > >>is needed due to that / is parent directory ?)
> > > >>
> > > >>Thank you
> > > >>Pavel
> > > >>
> > > >>On Tue, 12 Jul 2005 20:46:20 -0700 (PDT)
> > > >>Wilko Kroeger <[log in to unmask]> wrote:
> > > >>
> > > >>
> > > >>>Hello Pavel
> > > >>>
> > > >>>I suspect that this is a problem with the mss configuration. In the
> > > >>>dev. version the mss command is ignored if the path is only stage-able
> > > >>>and nodread, nocheck, nomig are specified.
> > > >>>Do you see in the xrdlog file the line
> > > >>> mssgwcmd ignored; no MSS paths present.
> > > >>>from the xrootd startup ?
> > > >>>
> > > >>>If this is the case you could add
> > > >>>oss.check
> > > >>>to your config which should cause xrootd to use the mssgwcmd command.
> > > >>>(oss.dread ot oss.rcreate should also do the trick). But if you
> > > >>>specify any of these options xrootd will behave a little bit different
> > > >>>from how it is configured now.
> > > >>>
> > > >>>The problem has been fixed in CVS and will be in the next release.
> > > >>>
> > > >>>
> > > >>>Cheers,
> > > >>> Wilko
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>On Tue, 12 Jul 2005, Pavel Jakl wrote:
> > > >>>
> > > >>>
> > > >>>>Hi for some time,
> > > >>>>
> > > >>>>Firstly, we've installed the latest development version of xrootd on
> > > >>>>100 nodes (1 redirector + 1 supervisor + 98 dataservers) with the
> > > >>>>support of open load balancing and everything is working great and
> > > >>>>super. I would like to say, brilliant work.
> > > >>>>
> > > >>>>To my problem, I want to enable the MSS interface. We've implemented
> > > >>>>scripts for performing meta-data operations and staging files from
> > > >>>>HPSS. (directives oss.mssgwcmd, oss.stagecmd). I 've inserted
> > > >>>>everything in config file for dataserver and tried to obtain a file
> > > >>>>from HPSS. In redirector log I found this error message:
> > > >>>>
> > > >>>> 050712 19:35:48 18366 odc_Locate: starlib.10447:13@rcas6230 given
> > > >>>> error msg 'No servers are available to read the file.' by xrdstar
> > > >>>> path=/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/s
> > > >>>> t_physics_5040131_raw_2010003.MuDst.root
> > > >>>>
> > > >>>>and found out that scripts was not even called once time. (this I
> > > >>>>know from debug support of my script). My question is, if I have
> > > >>>>something wrong in my configuration file or I forgot something to
> > > >>>>add or I skiped something in reading of documentation.
> > > >>>>
> > > >>>>I am enclosing a configuration file for dataserver.
> > > >>>>
> > > >>>>Thank you for any advice
> > > >>>>Pavel
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>
> > >
> > > --
> > > ,,,,,
> > > ( o o )
> > > --m---U---m--
> > > Jerome
> >
