It does (more or less), thank you.
But if you could also answer the previous Email/post,
it would be great.
Thanks,
Wilko Kroeger wrote:
> Hello Pavel
>
> On Fri, 15 Jul 2005, Pavel Jakl wrote:
>
>
>>Hello again,
>>
>> In the MPS manual and mps_* scripts, there are things like
>>
>>$Config{xfrcmd} = ($isMPS ? '/opt/xrootd/utils/xfrcmd'
>> : '/usr/etc/ooss/pftp_client');
>>
>> and similar switch for port, nodes, account name ... while both
>>seem to accept ftp-like commands. Could you please explain what
>>was the philosophy of having separate commands for oss/mps (but
>>using the same scripts) and why different port/nodes ?? This is
>>not entirely clear to us ...
>
>
> The ooss scripts are used at SLAC to manage the cache files systems.
> The mps scripts evolved out of the ooss scripts and they are kept
> backwards compatible, meaning that they would work with the ooss scripts
> installed in /usr/etc/ooss. The $isMPS basically decides if the MPS
> scripts that come with xrootd are used or the ooss scripts (installed in
> /usr/etc/ooss). This allows site that have ooss installed (like SLAC) to
> use the mps scripts.
>
>
>> Since stagecmd for oss is pftp and mps stagecmd is
>>xfrcmd, it makes us puzzled as per what xfrcmd does in addition
>>of pftp. This does not appear to be explained in the documentation.
>
>
> The '/opt/xrootd/utils/xfrcmd' should do the same thing that
> '/usr/etc/ooss/pftp_client' does for HPSS.
> Different sites use different MSS systems and even if they use the same
> system the access methods could be different. Therefore the mps scripts were
> written so that an xrootd site can provide their own implementation for
> mss access without modifying the mps scripts. If you are using pftp to access
> HPSS you could just call it from within /opt/xrootd/utils/xfrcmd.
>
> I hope this clarified your issues a littele bit.
>
> Cheers,
> Wilko
>
>
>>
>>Jerome & Pavel
>>
>>
>>On Thu, 14 Jul 2005 20:50:39 -0400
>>Jerome LAURET <[log in to unmask]> wrote:
>>
>>
>>> Hello Wilko (possibly Andy),
>>>
>>> Thanks already for your many answers which clarifies
>>>things quite a lot but we are not entirely sure of some details.
>>>What we are trying to do is the following (re-explained from
>>>the start, it may not be possible but ..)
>>>
>>>- first, we already have files located as (no typo, note
>>> the path "massage" here)
>>>
>>> /data?/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
>>>
>>> where ? stands for any number between 0 and 3. Each file
>>> correspond to its initial source (and continuing with our
>>> example)
>>>
>>> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>>>
>>> so, intrinsically, /data*/starlib/reco/... is equivalent to
>>> /home/starreco/reco/... (different PFN, same LFN and in our
>>> scheme, the path is slightly modified to fall back on our
>>> feet PFN -> LFN)
>>>
>>>
>>>- Our application takes advantage of already catalogued
>>> PFN. We would like to do a smooth transition as there is
>>> about 50+ TB of data already placed ... We nicely access
>>> the files using rootd referencing the files as
>>>root://node//data1/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
>>> as the files are a-piori known
>>>
>>>
>>>- Xrootd saves us a lot of course, for example, being able
>>> to reduce the name space to ONLY
>>>root://redirector//home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>>>
>>> as if we are accessing HPSS files while Xrootd handles it
>>> would be reat. BUT we also would like to be able to access
>>> the files the usual-current way i.e. within a syntax
>>> indicating //node/data1.
>>>
>>>- For this to happen, Xrootd would need to understand BOTH
>>> /home/starreco/reco using /data* as cache (for example i.e. MPS)
>>> AND also be able to access /data*/starlib/reco in direct
>>> access mode
>>>
>>> If so, several questions comes to mind
>>>
>>> /home/starreco would be populated with soft-links to files named
>>> /data?/%home%starreco%... which shares the space with the files
>>> already there.
>>>
>>> * Will Xrootd be happy with that ??? [a priori, why not]
>>> Seems like a possible space management nightmare to me
>>> (shared space)
>>>
>>> * What happen if in the /home/starreco/reco tree (local not
>>> HPSS) a soft-link is accidentely removed ?? Is it re-created ??
>>>
>>> * I assume the %XX%YY% files are managed i.e. deleted along
>>> the line of a policy. But folowing the above question, what
>>> happens if a file has disappeared but tyhe soft-link remains ??
>>> Is Xrootd smart enough to find it and re-stage ??
>>>
>>> * Are a lot of flat files in an identical /data?/ (i.e. files
>>> of the form %XX%YY% into a single dir) is read efficient ??
>>> Imagine that we get several 10k files in a single directory,
>>> will that cause a performance impact of some sort (directory
>>> lookup or otherwise) ??
>>>
>>> * By the way, does "readonly" have the meaning of 'cannot open
>>> a new file' or really readonly (which terminology seems to
>>> be incompatible with writting files into /data*). I assume
>>> the former.
>>>
>>>
>>> * Now for a controversial question. stagecmd command accepts
>>> in one of its form two arguments:
>>> HPSSPathAsInput localPath
>>> If localPath was only a "suggested path" and stagecmd would
>>> RETURN to Xrootd (as sstring) its final file placement,
>>> we could implement our own space management mechanism and
>>> policy. I can understand this not an item for the wish list
>>> but this WOULD have resolved our funny disk mapping and
>>> namespace issue [what it would not do is name space reduction
>>> at the end to a single namespace but that is another story].
>>> Comments ??
>>>
>>>- And Pavel would like to know if someone has an example of the
>>> /opt/xrootd/utils/xfrcmd script (to be sent to list or privately).
>>>
>>>
>>> Thanks & cheers,
>>>
>>>
>>>Wilko Kroeger wrote:
>>>
>>>>Hello Pavel
>>>>
>>>>Let me try to answer your questions.
>>>>
>>>>1) The stage command 'stagecmd':
>>>>
>>>>If a file is not on disk xrootd tries to stage the file onto disk. It
>>>>calls the stagecmd with two arguments:
>>>> stagecmd <remoteFileName> <localFileName>
>>>>where remoteFileName is the name in HPSS and localFileName is the one on
>>>>disk. Xrootd forms these names from the file name provide by the client
>>>>and the two prefixes oss.remoteroot and oss.localroot
>>>>For example:
>>>>the xrootd config file contains
>>>>oss.remoteroot /fr
>>>>oss.localroot /fl
>>>>a client requests a file like:
>>>> xrdcp root://dataserver//my/test.file test.file
>>>>in this case the stage command is called:
>>>> stagecmd /fr/my/test.file /fl/my/test.file
>>>>
>>>>If oss.remoteroot and oss.localroot are not specified the arguments to the
>>>>stagecmd is just the file name specified by the client.
>>>>
>>>>As you can see the files will always be staged into the same disk if you
>>>>use the oss.localroot. If you have more then one disk on an xrootd server
>>>>you want to use the cache system and a stagecmd that is aware of the cache
>>>>system.
>>>>
>>>>The xrootd actually comes with a cache aware stage command. You can find
>>>>it in the utils directory of an xrootd release it is called mps_Stage.
>>>>I haven't used it myself but I will find out how to use it.
>>>>The utils dir contains a few mps_XXX utils that are used to manage a cache
>>>>system. On the xrootd web site there is a document that describes the mps
>>>>system: http://xrootd.slac.stanford.edu/doc/mps_config/mps_config.htm
>>>>
>>>>
>>>>2) The cache file system:
>>>>
>>>>A file that is staged into a cache file system is physically put in any of
>>>>the specified caches and a link between this files and the proper file
>>>>name is created. For example:
>>>>
>>>>Lets assume you have the following file:
>>>> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>>>>and it is in the cache /data3:
>>>>
>>>>
>>>>
>>>>>ls -l /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>>>>
>>>>would show:
>>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root ->
>>>> /data3/%home%starreco%reco%productionHigh%FullField%P04ik%2004%f1.root
>>>>
>>>>As I mentioned above, if you want to use a cache system the stagecmd
>>>>has to be aware of it.
>>>>
>>>>3) Your setup:
>>>>
>>>>In the configuration that you described below you export all the cache
>>>>system (/dataN) due to the olbd.path and thats maybe something you don't
>>>>want to do. This also means the client has to access files with the
>>>>name /data3/home/... making your system setup visible to the client.
>>>>
>>>>Instead, it seems to me you that would like to make files with the name
>>>>'/home/....' accessible to users but on the xrootd server these files are
>>>>stored in /dataN/... .
>>>>
>>>>You could configure your system with:
>>>>
>>>>oss.path /home dread nomig stage
>>>>
>>>>oss.cache /data*
>>>>
>>>>xrootd.export /home
>>>>
>>>>You also have to specify the stagecmd and the mssgwd command. The stage
>>>>command you obviously need in order to get a file out of HPSS, and the
>>>>mssgwd command is needed because xrootd first checks if a file is present
>>>>in HPSS. If you don't want xrootd to do this you could provide a
>>>>implementation that returns dummy data.
>>>>I could provide you with a little dummy script that implements some of the
>>>>mssgwd commands.
>>>>
>>>>
>>>>I don't now why all your path are migratable (mig). I suspect that one of
>>>>the options forces to turn on mig but I don't know which one. I have to
>>>>look into this.
>>>>
>>>>
>>>>The
>>>>oss.path / .....
>>>>is always there by default.
>>>>
>>>>I hope I clarified some of your questions. Let me know if not.
>>>>
>>>>Cheers,
>>>> Wilko
>>>>
>>>>
>>>>
>>>>On Wed, 13 Jul 2005, Pavel Jakl wrote:
>>>>
>>>>
>>>>
>>>>>Hi Wilko,
>>>>>
>>>>>many thanks for advice. With directive oss.check client is redirected to
>>>>>other node and staging of requested file is started.
>>>>>I have some other questions, if you can help me, it will be super.
>>>>>
>>>>>I want to stage a file from HPSS for example with path like:
>>>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
>>>>>5040131_raw_2010003.MuDst.root
>>>>>
>>>>>and copy to path /data0 or /data1 or /data2 or /data3 which I've
>>>>>specified with directives:
>>>>>
>>>>>oss.path /data0 dread nomig stage
>>>>>oss.path /data1 dread nomig stage
>>>>>oss.path /data2 dread nomig stage
>>>>>oss.path /data3 dread nomig stage
>>>>>
>>>>>My question is if my thoughts are right and file with HPPS path:
>>>>>
>>>>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
>>>>>5040131_raw_2010003.MuDst.root
>>>>>
>>>>>will be staged to local path:
>>>>>
>>>>>/data0/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_ph
>>>>>ysics_5040131_raw_2010003.MuDst.root
>>>>>i.e that /data0 is path to ve pre-pended to local path. Or is there some
>>>>>other directive, which can help me to do that ?
>>>>>
>>>>>My second question is if I need to have specified oss.mssgwcmd command
>>>>>when I don't plan to migrate any files from local disk to HPSS ?
>>>>>
>>>>>The third question is about oss.cache directive. Can I have the location
>>>>>of cache in the same directory as I have the files to export ?
>>>>>For example, I have all files in directories:
>>>>>/data0, /data1, /data2 or /data3 (i export with directive xrootd.export)
>>>>>and can I have a oss.cache /data*.
>>>>>
>>>>>My fourth problem is with oss.path again, I've specified this:
>>>>>oss.path /data0 dread nomig stage
>>>>>oss.path /data1 dread nomig stage
>>>>>oss.path /data2 dread nomig stage
>>>>>oss.path /data3 dread nomig stage
>>>>>
>>>>>but in log file is this:
>>>>>oss.path /data3 r/o check dread mig nomkeep nomlock nommap norcreate
>>>>>stage
>>>>>oss.path /data2 r/o check dread mig nomkeep nomlock nommap norcreate
>>>>>stage
>>>>>oss.path /data1 r/o check dread mig nomkeep nomlock nommap norcreate
>>>>>stage
>>>>>oss.path /data0 r/o check dread mig nomkeep nomlock nommap norcreate
>>>>>stage
>>>>>
>>>>>as you can see with mig directive and in addation is here a line:
>>>>>
>>>>>oss.path / r/o check dread mig nomkeep nomlock nommap norcreate stage
>>>>>
>>>>>Is this correct ? Becaouse I dont want to stage any files to path /. (Or
>>>>>is needed due to that / is parent directory ?)
>>>>>
>>>>>Thank you
>>>>>Pavel
>>>>>
>>>>>On Tue, 12 Jul 2005 20:46:20 -0700 (PDT)
>>>>>Wilko Kroeger <[log in to unmask]> wrote:
>>>>>
>>>>>
>>>>>
>>>>>>Hello Pavel
>>>>>>
>>>>>>I suspect that this is a problem with the mss configuration. In the
>>>>>>dev. version the mss command is ignored if the path is only stage-able
>>>>>>and nodread, nocheck, nomig are specified.
>>>>>>Do you see in the xrdlog file the line
>>>>>> mssgwcmd ignored; no MSS paths present.
>>>>>
>>>>>>from the xrootd startup ?
>>>>>
>>>>>>If this is the case you could add
>>>>>>oss.check
>>>>>>to your config which should cause xrootd to use the mssgwcmd command.
>>>>>>(oss.dread ot oss.rcreate should also do the trick). But if you
>>>>>>specify any of these options xrootd will behave a little bit different
>>>>>
>>>>>>from how it is configured now.
>>>>>
>>>>>>The problem has been fixed in CVS and will be in the next release.
>>>>>>
>>>>>>
>>>>>>Cheers,
>>>>>> Wilko
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>On Tue, 12 Jul 2005, Pavel Jakl wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Hi for some time,
>>>>>>>
>>>>>>>Firstly, we've installed the latest development version of xrootd on
>>>>>>>100 nodes (1 redirector + 1 supervisor + 98 dataservers) with the
>>>>>>>support of open load balancing and everything is working great and
>>>>>>>super. I would like to say, brilliant work.
>>>>>>>
>>>>>>>To my problem, I want to enable the MSS interface. We've implemented
>>>>>>>scripts for performing meta-data operations and staging files from
>>>>>>>HPSS. (directives oss.mssgwcmd, oss.stagecmd). I 've inserted
>>>>>>>everything in config file for dataserver and tried to obtain a file
>>>>>>
>>>>>>>from HPSS. In redirector log I found this error message:
>>>>>>
>>>>>>>050712 19:35:48 18366 odc_Locate: starlib.10447:13@rcas6230 given
>>>>>>>error msg 'No servers are available to read the file.' by xrdstar
>>>>>>>path=/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/s
>>>>>>>t_physics_5040131_raw_2010003.MuDst.root
>>>>>>>
>>>>>>>and found out that scripts was not even called once time. (this I
>>>>>>>know from debug support of my script). My question is, if I have
>>>>>>>something wrong in my configuration file or I forgot something to
>>>>>>>add or I skiped something in reading of documentation.
>>>>>>>
>>>>>>>I am enclosing a configuration file for dataserver.
>>>>>>>
>>>>>>>Thank you for any advice
>>>>>>>Pavel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>--
>>> ,,,,,
>>> ( o o )
>>> --m---U---m--
>>> Jerome
>>
--
,,,,,
( o o )
--m---U---m--
Jerome
