Hello Jerome
I am just trying something out related to your setup
but run into some trouble. Anyhow, I will answer the
previous Email soon.
Cheers,
Wilko
On Fri, 15 Jul 2005, Jerome LAURET wrote:
>
> It does (more or less), thank you.
>
> But if you could also answer the previous Email/post,
> it would be great.
>
> Thanks,
>
> Wilko Kroeger wrote:
> > Hello Pavel
> >
> > On Fri, 15 Jul 2005, Pavel Jakl wrote:
> >
> >
> >>Hello again,
> >>
> >> In the MPS manual and mps_* scripts, there are things like
> >>
> >>$Config{xfrcmd} = ($isMPS ? '/opt/xrootd/utils/xfrcmd'
> >> : '/usr/etc/ooss/pftp_client');
> >>
> >> and similar switch for port, nodes, account name ... while both
> >>seem to accept ftp-like commands. Could you please explain what
> >>was the philosophy of having separate commands for oss/mps (but
> >>using the same scripts) and why different port/nodes ?? This is
> >>not entirely clear to us ...
> >
> >
> > The ooss scripts are used at SLAC to manage the cache files systems.
> > The mps scripts evolved out of the ooss scripts and they are kept
> > backwards compatible, meaning that they would work with the ooss scripts
> > installed in /usr/etc/ooss. The $isMPS basically decides if the MPS
> > scripts that come with xrootd are used or the ooss scripts (installed in
> > /usr/etc/ooss). This allows site that have ooss installed (like SLAC) to
> > use the mps scripts.
> >
> >
> >> Since stagecmd for oss is pftp and mps stagecmd is
> >>xfrcmd, it makes us puzzled as per what xfrcmd does in addition
> >>of pftp. This does not appear to be explained in the documentation.
> >
> >
> > The '/opt/xrootd/utils/xfrcmd' should do the same thing that
> > '/usr/etc/ooss/pftp_client' does for HPSS.
> > Different sites use different MSS systems and even if they use the same
> > system the access methods could be different. Therefore the mps scripts were
> > written so that an xrootd site can provide their own implementation for
> > mss access without modifying the mps scripts. If you are using pftp to access
> > HPSS you could just call it from within /opt/xrootd/utils/xfrcmd.
> >
> > I hope this clarified your issues a littele bit.
> >
> > Cheers,
> > Wilko
> >
> >
> >>
> >>Jerome & Pavel
> >>
> >>
> >>On Thu, 14 Jul 2005 20:50:39 -0400
> >>Jerome LAURET <[log in to unmask]> wrote:
> >>
> >>
> >>> Hello Wilko (possibly Andy),
> >>>
> >>> Thanks already for your many answers which clarifies
> >>>things quite a lot but we are not entirely sure of some details.
> >>>What we are trying to do is the following (re-explained from
> >>>the start, it may not be possible but ..)
> >>>
> >>>- first, we already have files located as (no typo, note
> >>> the path "massage" here)
> >>>
> >>> /data?/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>>
> >>> where ? stands for any number between 0 and 3. Each file
> >>> correspond to its initial source (and continuing with our
> >>> example)
> >>>
> >>> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>>
> >>> so, intrinsically, /data*/starlib/reco/... is equivalent to
> >>> /home/starreco/reco/... (different PFN, same LFN and in our
> >>> scheme, the path is slightly modified to fall back on our
> >>> feet PFN -> LFN)
> >>>
> >>>
> >>>- Our application takes advantage of already catalogued
> >>> PFN. We would like to do a smooth transition as there is
> >>> about 50+ TB of data already placed ... We nicely access
> >>> the files using rootd referencing the files as
> >>>root://node//data1/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>> as the files are a-piori known
> >>>
> >>>
> >>>- Xrootd saves us a lot of course, for example, being able
> >>> to reduce the name space to ONLY
> >>>root://redirector//home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>>
> >>> as if we are accessing HPSS files while Xrootd handles it
> >>> would be reat. BUT we also would like to be able to access
> >>> the files the usual-current way i.e. within a syntax
> >>> indicating //node/data1.
> >>>
> >>>- For this to happen, Xrootd would need to understand BOTH
> >>> /home/starreco/reco using /data* as cache (for example i.e. MPS)
> >>> AND also be able to access /data*/starlib/reco in direct
> >>> access mode
> >>>
> >>> If so, several questions comes to mind
> >>>
> >>> /home/starreco would be populated with soft-links to files named
> >>> /data?/%home%starreco%... which shares the space with the files
> >>> already there.
> >>>
> >>> * Will Xrootd be happy with that ??? [a priori, why not]
> >>> Seems like a possible space management nightmare to me
> >>> (shared space)
> >>>
> >>> * What happen if in the /home/starreco/reco tree (local not
> >>> HPSS) a soft-link is accidentely removed ?? Is it re-created ??
> >>>
> >>> * I assume the %XX%YY% files are managed i.e. deleted along
> >>> the line of a policy. But folowing the above question, what
> >>> happens if a file has disappeared but tyhe soft-link remains ??
> >>> Is Xrootd smart enough to find it and re-stage ??
> >>>
> >>> * Are a lot of flat files in an identical /data?/ (i.e. files
> >>> of the form %XX%YY% into a single dir) is read efficient ??
> >>> Imagine that we get several 10k files in a single directory,
> >>> will that cause a performance impact of some sort (directory
> >>> lookup or otherwise) ??
> >>>
> >>> * By the way, does "readonly" have the meaning of 'cannot open
> >>> a new file' or really readonly (which terminology seems to
> >>> be incompatible with writting files into /data*). I assume
> >>> the former.
> >>>
> >>>
> >>> * Now for a controversial question. stagecmd command accepts
> >>> in one of its form two arguments:
> >>> HPSSPathAsInput localPath
> >>> If localPath was only a "suggested path" and stagecmd would
> >>> RETURN to Xrootd (as sstring) its final file placement,
> >>> we could implement our own space management mechanism and
> >>> policy. I can understand this not an item for the wish list
> >>> but this WOULD have resolved our funny disk mapping and
> >>> namespace issue [what it would not do is name space reduction
> >>> at the end to a single namespace but that is another story].
> >>> Comments ??
> >>>
> >>>- And Pavel would like to know if someone has an example of the
> >>> /opt/xrootd/utils/xfrcmd script (to be sent to list or privately).
> >>>
> >>>
> >>> Thanks & cheers,
> >>>
> >>>
> >>>Wilko Kroeger wrote:
> >>>
> >>>>Hello Pavel
> >>>>
> >>>>Let me try to answer your questions.
> >>>>
> >>>>1) The stage command 'stagecmd':
> >>>>
> >>>>If a file is not on disk xrootd tries to stage the file onto disk. It
> >>>>calls the stagecmd with two arguments:
> >>>> stagecmd <remoteFileName> <localFileName>
> >>>>where remoteFileName is the name in HPSS and localFileName is the one on
> >>>>disk. Xrootd forms these names from the file name provide by the client
> >>>>and the two prefixes oss.remoteroot and oss.localroot
> >>>>For example:
> >>>>the xrootd config file contains
> >>>>oss.remoteroot /fr
> >>>>oss.localroot /fl
> >>>>a client requests a file like:
> >>>> xrdcp root://dataserver//my/test.file test.file
> >>>>in this case the stage command is called:
> >>>> stagecmd /fr/my/test.file /fl/my/test.file
> >>>>
> >>>>If oss.remoteroot and oss.localroot are not specified the arguments to the
> >>>>stagecmd is just the file name specified by the client.
> >>>>
> >>>>As you can see the files will always be staged into the same disk if you
> >>>>use the oss.localroot. If you have more then one disk on an xrootd server
> >>>>you want to use the cache system and a stagecmd that is aware of the cache
> >>>>system.
> >>>>
> >>>>The xrootd actually comes with a cache aware stage command. You can find
> >>>>it in the utils directory of an xrootd release it is called mps_Stage.
> >>>>I haven't used it myself but I will find out how to use it.
> >>>>The utils dir contains a few mps_XXX utils that are used to manage a cache
> >>>>system. On the xrootd web site there is a document that describes the mps
> >>>>system: http://xrootd.slac.stanford.edu/doc/mps_config/mps_config.htm
> >>>>
> >>>>
> >>>>2) The cache file system:
> >>>>
> >>>>A file that is staged into a cache file system is physically put in any of
> >>>>the specified caches and a link between this files and the proper file
> >>>>name is created. For example:
> >>>>
> >>>>Lets assume you have the following file:
> >>>> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>>>and it is in the cache /data3:
> >>>>
> >>>>
> >>>>
> >>>>>ls -l /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> >>>>
> >>>>would show:
> >>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root ->
> >>>> /data3/%home%starreco%reco%productionHigh%FullField%P04ik%2004%f1.root
> >>>>
> >>>>As I mentioned above, if you want to use a cache system the stagecmd
> >>>>has to be aware of it.
> >>>>
> >>>>3) Your setup:
> >>>>
> >>>>In the configuration that you described below you export all the cache
> >>>>system (/dataN) due to the olbd.path and thats maybe something you don't
> >>>>want to do. This also means the client has to access files with the
> >>>>name /data3/home/... making your system setup visible to the client.
> >>>>
> >>>>Instead, it seems to me you that would like to make files with the name
> >>>>'/home/....' accessible to users but on the xrootd server these files are
> >>>>stored in /dataN/... .
> >>>>
> >>>>You could configure your system with:
> >>>>
> >>>>oss.path /home dread nomig stage
> >>>>
> >>>>oss.cache /data*
> >>>>
> >>>>xrootd.export /home
> >>>>
> >>>>You also have to specify the stagecmd and the mssgwd command. The stage
> >>>>command you obviously need in order to get a file out of HPSS, and the
> >>>>mssgwd command is needed because xrootd first checks if a file is present
> >>>>in HPSS. If you don't want xrootd to do this you could provide a
> >>>>implementation that returns dummy data.
> >>>>I could provide you with a little dummy script that implements some of the
> >>>>mssgwd commands.
> >>>>
> >>>>
> >>>>I don't now why all your path are migratable (mig). I suspect that one of
> >>>>the options forces to turn on mig but I don't know which one. I have to
> >>>>look into this.
> >>>>
> >>>>
> >>>>The
> >>>>oss.path / .....
> >>>>is always there by default.
> >>>>
> >>>>I hope I clarified some of your questions. Let me know if not.
> >>>>
> >>>>Cheers,
> >>>> Wilko
> >>>>
> >>>>
> >>>>
> >>>>On Wed, 13 Jul 2005, Pavel Jakl wrote:
> >>>>
> >>>>
> >>>>
> >>>>>Hi Wilko,
> >>>>>
> >>>>>many thanks for advice. With directive oss.check client is redirected to
> >>>>>other node and staging of requested file is started.
> >>>>>I have some other questions, if you can help me, it will be super.
> >>>>>
> >>>>>I want to stage a file from HPSS for example with path like:
> >>>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> >>>>>5040131_raw_2010003.MuDst.root
> >>>>>
> >>>>>and copy to path /data0 or /data1 or /data2 or /data3 which I've
> >>>>>specified with directives:
> >>>>>
> >>>>>oss.path /data0 dread nomig stage
> >>>>>oss.path /data1 dread nomig stage
> >>>>>oss.path /data2 dread nomig stage
> >>>>>oss.path /data3 dread nomig stage
> >>>>>
> >>>>>My question is if my thoughts are right and file with HPPS path:
> >>>>>
> >>>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
> >>>>>5040131_raw_2010003.MuDst.root
> >>>>>
> >>>>>will be staged to local path:
> >>>>>
> >>>>>/data0/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_ph
> >>>>>ysics_5040131_raw_2010003.MuDst.root
> >>>>>i.e that /data0 is path to ve pre-pended to local path. Or is there some
> >>>>>other directive, which can help me to do that ?
> >>>>>
> >>>>>My second question is if I need to have specified oss.mssgwcmd command
> >>>>>when I don't plan to migrate any files from local disk to HPSS ?
> >>>>>
> >>>>>The third question is about oss.cache directive. Can I have the location
> >>>>>of cache in the same directory as I have the files to export ?
> >>>>>For example, I have all files in directories:
> >>>>>/data0, /data1, /data2 or /data3 (i export with directive xrootd.export)
> >>>>>and can I have a oss.cache /data*.
> >>>>>
> >>>>>My fourth problem is with oss.path again, I've specified this:
> >>>>>oss.path /data0 dread nomig stage
> >>>>>oss.path /data1 dread nomig stage
> >>>>>oss.path /data2 dread nomig stage
> >>>>>oss.path /data3 dread nomig stage
> >>>>>
> >>>>>but in log file is this:
> >>>>>oss.path /data3 r/o check dread mig nomkeep nomlock nommap norcreate
> >>>>>stage
> >>>>>oss.path /data2 r/o check dread mig nomkeep nomlock nommap norcreate
> >>>>>stage
> >>>>>oss.path /data1 r/o check dread mig nomkeep nomlock nommap norcreate
> >>>>>stage
> >>>>>oss.path /data0 r/o check dread mig nomkeep nomlock nommap norcreate
> >>>>>stage
> >>>>>
> >>>>>as you can see with mig directive and in addation is here a line:
> >>>>>
> >>>>>oss.path / r/o check dread mig nomkeep nomlock nommap norcreate stage
> >>>>>
> >>>>>Is this correct ? Becaouse I dont want to stage any files to path /. (Or
> >>>>>is needed due to that / is parent directory ?)
> >>>>>
> >>>>>Thank you
> >>>>>Pavel
> >>>>>
> >>>>>On Tue, 12 Jul 2005 20:46:20 -0700 (PDT)
> >>>>>Wilko Kroeger <[log in to unmask]> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Hello Pavel
> >>>>>>
> >>>>>>I suspect that this is a problem with the mss configuration. In the
> >>>>>>dev. version the mss command is ignored if the path is only stage-able
> >>>>>>and nodread, nocheck, nomig are specified.
> >>>>>>Do you see in the xrdlog file the line
> >>>>>> mssgwcmd ignored; no MSS paths present.
> >>>>>
> >>>>>>from the xrootd startup ?
> >>>>>
> >>>>>>If this is the case you could add
> >>>>>>oss.check
> >>>>>>to your config which should cause xrootd to use the mssgwcmd command.
> >>>>>>(oss.dread ot oss.rcreate should also do the trick). But if you
> >>>>>>specify any of these options xrootd will behave a little bit different
> >>>>>
> >>>>>>from how it is configured now.
> >>>>>
> >>>>>>The problem has been fixed in CVS and will be in the next release.
> >>>>>>
> >>>>>>
> >>>>>>Cheers,
> >>>>>> Wilko
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>On Tue, 12 Jul 2005, Pavel Jakl wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Hi for some time,
> >>>>>>>
> >>>>>>>Firstly, we've installed the latest development version of xrootd on
> >>>>>>>100 nodes (1 redirector + 1 supervisor + 98 dataservers) with the
> >>>>>>>support of open load balancing and everything is working great and
> >>>>>>>super. I would like to say, brilliant work.
> >>>>>>>
> >>>>>>>To my problem, I want to enable the MSS interface. We've implemented
> >>>>>>>scripts for performing meta-data operations and staging files from
> >>>>>>>HPSS. (directives oss.mssgwcmd, oss.stagecmd). I 've inserted
> >>>>>>>everything in config file for dataserver and tried to obtain a file
> >>>>>>
> >>>>>>>from HPSS. In redirector log I found this error message:
> >>>>>>
> >>>>>>>050712 19:35:48 18366 odc_Locate: starlib.10447:13@rcas6230 given
> >>>>>>>error msg 'No servers are available to read the file.' by xrdstar
> >>>>>>>path=/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/s
> >>>>>>>t_physics_5040131_raw_2010003.MuDst.root
> >>>>>>>
> >>>>>>>and found out that scripts was not even called once time. (this I
> >>>>>>>know from debug support of my script). My question is, if I have
> >>>>>>>something wrong in my configuration file or I forgot something to
> >>>>>>>add or I skiped something in reading of documentation.
> >>>>>>>
> >>>>>>>I am enclosing a configuration file for dataserver.
> >>>>>>>
> >>>>>>>Thank you for any advice
> >>>>>>>Pavel
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>--
> >>> ,,,,,
> >>> ( o o )
> >>> --m---U---m--
> >>> Jerome
> >>
>
> --
> ,,,,,
> ( o o )
> --m---U---m--
> Jerome
>
