Hello Wilko (possibly Andy),
Thanks already for your many answers which clarifies
things quite a lot but we are not entirely sure of some details.
What we are trying to do is the following (re-explained from
the start, it may not be possible but ..)
- first, we already have files located as (no typo, note
the path "massage" here)
/data?/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
where ? stands for any number between 0 and 3. Each file
correspond to its initial source (and continuing with our
example)
/home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
so, intrinsically, /data*/starlib/reco/... is equivalent to
/home/starreco/reco/... (different PFN, same LFN and in our
scheme, the path is slightly modified to fall back on our
feet PFN -> LFN)
- Our application takes advantage of already catalogued
PFN. We would like to do a smooth transition as there is
about 50+ TB of data already placed ... We nicely access
the files using rootd referencing the files as
root://node//data1/starlib/reco/productionHigh/FullField/P04ik/2004/f1.root
as the files are a-piori known
- Xrootd saves us a lot of course, for example, being able
to reduce the name space to ONLY
root://redirector//home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
as if we are accessing HPSS files while Xrootd handles it
would be reat. BUT we also would like to be able to access
the files the usual-current way i.e. within a syntax
indicating //node/data1.
- For this to happen, Xrootd would need to understand BOTH
/home/starreco/reco using /data* as cache (for example i.e. MPS)
AND also be able to access /data*/starlib/reco in direct
access mode
If so, several questions comes to mind
/home/starreco would be populated with soft-links to files named
/data?/%home%starreco%... which shares the space with the files
already there.
* Will Xrootd be happy with that ??? [a priori, why not]
Seems like a possible space management nightmare to me
(shared space)
* What happen if in the /home/starreco/reco tree (local not
HPSS) a soft-link is accidentely removed ?? Is it re-created ??
* I assume the %XX%YY% files are managed i.e. deleted along
the line of a policy. But folowing the above question, what
happens if a file has disappeared but tyhe soft-link remains ??
Is Xrootd smart enough to find it and re-stage ??
* Are a lot of flat files in an identical /data?/ (i.e. files
of the form %XX%YY% into a single dir) is read efficient ??
Imagine that we get several 10k files in a single directory,
will that cause a performance impact of some sort (directory
lookup or otherwise) ??
* By the way, does "readonly" have the meaning of 'cannot open
a new file' or really readonly (which terminology seems to
be incompatible with writting files into /data*). I assume
the former.
* Now for a controversial question. stagecmd command accepts
in one of its form two arguments:
HPSSPathAsInput localPath
If localPath was only a "suggested path" and stagecmd would
RETURN to Xrootd (as sstring) its final file placement,
we could implement our own space management mechanism and
policy. I can understand this not an item for the wish list
but this WOULD have resolved our funny disk mapping and
namespace issue [what it would not do is name space reduction
at the end to a single namespace but that is another story].
Comments ??
- And Pavel would like to know if someone has an example of the
/opt/xrootd/utils/xfrcmd script (to be sent to list or privately).
Thanks & cheers,
Wilko Kroeger wrote:
> Hello Pavel
>
> Let me try to answer your questions.
>
> 1) The stage command 'stagecmd':
>
> If a file is not on disk xrootd tries to stage the file onto disk. It
> calls the stagecmd with two arguments:
> stagecmd <remoteFileName> <localFileName>
> where remoteFileName is the name in HPSS and localFileName is the one on
> disk. Xrootd forms these names from the file name provide by the client
> and the two prefixes oss.remoteroot and oss.localroot
> For example:
> the xrootd config file contains
> oss.remoteroot /fr
> oss.localroot /fl
> a client requests a file like:
> xrdcp root://dataserver//my/test.file test.file
> in this case the stage command is called:
> stagecmd /fr/my/test.file /fl/my/test.file
>
> If oss.remoteroot and oss.localroot are not specified the arguments to the
> stagecmd is just the file name specified by the client.
>
> As you can see the files will always be staged into the same disk if you
> use the oss.localroot. If you have more then one disk on an xrootd server
> you want to use the cache system and a stagecmd that is aware of the cache
> system.
>
> The xrootd actually comes with a cache aware stage command. You can find
> it in the utils directory of an xrootd release it is called mps_Stage.
> I haven't used it myself but I will find out how to use it.
> The utils dir contains a few mps_XXX utils that are used to manage a cache
> system. On the xrootd web site there is a document that describes the mps
> system: http://xrootd.slac.stanford.edu/doc/mps_config/mps_config.htm
>
>
> 2) The cache file system:
>
> A file that is staged into a cache file system is physically put in any of
> the specified caches and a link between this files and the proper file
> name is created. For example:
>
> Lets assume you have the following file:
> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
> and it is in the cache /data3:
>
>
>>ls -l /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root
>
> would show:
> /home/starreco/reco/productionHigh/FullField/P04ik/2004/f1.root ->
> /data3/%home%starreco%reco%productionHigh%FullField%P04ik%2004%f1.root
>
> As I mentioned above, if you want to use a cache system the stagecmd
> has to be aware of it.
>
> 3) Your setup:
>
> In the configuration that you described below you export all the cache
> system (/dataN) due to the olbd.path and thats maybe something you don't
> want to do. This also means the client has to access files with the
> name /data3/home/... making your system setup visible to the client.
>
> Instead, it seems to me you that would like to make files with the name
> '/home/....' accessible to users but on the xrootd server these files are
> stored in /dataN/... .
>
> You could configure your system with:
>
> oss.path /home dread nomig stage
>
> oss.cache /data*
>
> xrootd.export /home
>
> You also have to specify the stagecmd and the mssgwd command. The stage
> command you obviously need in order to get a file out of HPSS, and the
> mssgwd command is needed because xrootd first checks if a file is present
> in HPSS. If you don't want xrootd to do this you could provide a
> implementation that returns dummy data.
> I could provide you with a little dummy script that implements some of the
> mssgwd commands.
>
>
> I don't now why all your path are migratable (mig). I suspect that one of
> the options forces to turn on mig but I don't know which one. I have to
> look into this.
>
>
> The
> oss.path / .....
> is always there by default.
>
> I hope I clarified some of your questions. Let me know if not.
>
> Cheers,
> Wilko
>
>
>
> On Wed, 13 Jul 2005, Pavel Jakl wrote:
>
>
>>Hi Wilko,
>>
>>many thanks for advice. With directive oss.check client is redirected to
>>other node and staging of requested file is started.
>>I have some other questions, if you can help me, it will be super.
>>
>>I want to stage a file from HPSS for example with path like:
>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
>>5040131_raw_2010003.MuDst.root
>>
>>and copy to path /data0 or /data1 or /data2 or /data3 which I've
>>specified with directives:
>>
>>oss.path /data0 dread nomig stage
>>oss.path /data1 dread nomig stage
>>oss.path /data2 dread nomig stage
>>oss.path /data3 dread nomig stage
>>
>>My question is if my thoughts are right and file with HPPS path:
>>
>>/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_physics_
>>5040131_raw_2010003.MuDst.root
>>
>>will be staged to local path:
>>
>>/data0/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/st_ph
>>ysics_5040131_raw_2010003.MuDst.root
>>i.e that /data0 is path to ve pre-pended to local path. Or is there some
>>other directive, which can help me to do that ?
>>
>>My second question is if I need to have specified oss.mssgwcmd command
>>when I don't plan to migrate any files from local disk to HPSS ?
>>
>>The third question is about oss.cache directive. Can I have the location
>>of cache in the same directory as I have the files to export ?
>>For example, I have all files in directories:
>>/data0, /data1, /data2 or /data3 (i export with directive xrootd.export)
>>and can I have a oss.cache /data*.
>>
>>My fourth problem is with oss.path again, I've specified this:
>>oss.path /data0 dread nomig stage
>>oss.path /data1 dread nomig stage
>>oss.path /data2 dread nomig stage
>>oss.path /data3 dread nomig stage
>>
>>but in log file is this:
>>oss.path /data3 r/o check dread mig nomkeep nomlock nommap norcreate
>>stage
>>oss.path /data2 r/o check dread mig nomkeep nomlock nommap norcreate
>>stage
>>oss.path /data1 r/o check dread mig nomkeep nomlock nommap norcreate
>>stage
>>oss.path /data0 r/o check dread mig nomkeep nomlock nommap norcreate
>>stage
>>
>>as you can see with mig directive and in addation is here a line:
>>
>>oss.path / r/o check dread mig nomkeep nomlock nommap norcreate stage
>>
>>Is this correct ? Becaouse I dont want to stage any files to path /. (Or
>>is needed due to that / is parent directory ?)
>>
>>Thank you
>>Pavel
>>
>>On Tue, 12 Jul 2005 20:46:20 -0700 (PDT)
>>Wilko Kroeger <[log in to unmask]> wrote:
>>
>>
>>>Hello Pavel
>>>
>>>I suspect that this is a problem with the mss configuration. In the
>>>dev. version the mss command is ignored if the path is only stage-able
>>>and nodread, nocheck, nomig are specified.
>>>Do you see in the xrdlog file the line
>>> mssgwcmd ignored; no MSS paths present.
>>>from the xrootd startup ?
>>>
>>>If this is the case you could add
>>>oss.check
>>>to your config which should cause xrootd to use the mssgwcmd command.
>>>(oss.dread ot oss.rcreate should also do the trick). But if you
>>>specify any of these options xrootd will behave a little bit different
>>>from how it is configured now.
>>>
>>>The problem has been fixed in CVS and will be in the next release.
>>>
>>>
>>>Cheers,
>>> Wilko
>>>
>>>
>>>
>>>
>>>
>>>On Tue, 12 Jul 2005, Pavel Jakl wrote:
>>>
>>>
>>>>Hi for some time,
>>>>
>>>>Firstly, we've installed the latest development version of xrootd on
>>>>100 nodes (1 redirector + 1 supervisor + 98 dataservers) with the
>>>>support of open load balancing and everything is working great and
>>>>super. I would like to say, brilliant work.
>>>>
>>>>To my problem, I want to enable the MSS interface. We've implemented
>>>>scripts for performing meta-data operations and staging files from
>>>>HPSS. (directives oss.mssgwcmd, oss.stagecmd). I 've inserted
>>>>everything in config file for dataserver and tried to obtain a file
>>>>from HPSS. In redirector log I found this error message:
>>>>
>>>> 050712 19:35:48 18366 odc_Locate: starlib.10447:13@rcas6230 given
>>>> error msg 'No servers are available to read the file.' by xrdstar
>>>> path=/home/starreco/reco/productionHigh/FullField/P04ik/2004/040/s
>>>> t_physics_5040131_raw_2010003.MuDst.root
>>>>
>>>>and found out that scripts was not even called once time. (this I
>>>>know from debug support of my script). My question is, if I have
>>>>something wrong in my configuration file or I forgot something to
>>>>add or I skiped something in reading of documentation.
>>>>
>>>>I am enclosing a configuration file for dataserver.
>>>>
>>>>Thank you for any advice
>>>>Pavel
>>>>
>>>>
>>>>
>>
--
,,,,,
( o o )
--m---U---m--
Jerome
|