 |
 |
Hi Stephen Yes, I can reproduce the problem in analysis-24. However now that we have staging working it is less of an issue as we should no longer see users being asked to wait unessarily. Of course there may be other issues that we have not anticipated. cheers Manny Stephen J. Gowdy wrote: > Hi Tim, > It is possible for us to use our own version if absolutely > required. So you reproduce the problem in analysis-24? I don't know when > the bug was fixed. > > regards, > > Stephen. > > On Fri, 10 Jun 2005, Adye, TJ (Tim) wrote: > > >>Hi Stephen, >> >>This sounds very worrying. Does the fact that it's part of ROOT mean >>that there's no way to fix this problem for people using, eg., >>analysis-24? I can't see any version message from XTNetFile in my >>BetaMiniApp output, but I may have missed it among all the rest of the >>dross^H^H^H^H^H useful messages. >> >>Which version of ROOT fixed this problem? >> >>Tim. >> >> >>>-----Original Message----- >>>From: Stephen J. Gowdy [mailto:[log in to unmask]] >>>Sent: 10 June 2005 12:32 >>>To: Adye, TJ (Tim) >>>Cc: Andrew Hanushevsky; Fabrizio Furano; [log in to unmask]; >>>[log in to unmask]; Brew, CAJ (Chris); Bill Weeks; >>>[log in to unmask] >>>Subject: RE: PreStage Problems >>> >>>In recent releases it comes with ROOT. In older releases it >>>would print the version number when it opens the file. >>> >>>On Fri, 10 Jun 2005, Adye, TJ (Tim) wrote: >>> >>> >>>>Hi Fabrizio and Pete, >>>> >>>>Andrew Hanushevsky <[log in to unmask]> wrote: >>>> >>>> >>>>>The "continuing to hang" problem is a client problem. Here >>>>>the client is always asking for a cache refresh. So, either >>>>>an old client is being used (old clients had this bug and it >>>>>was fixed about 6 months ago) or the bug has returned under >>>>>this new scenario (I suspect that the latter is true). >>>> >>>>How can we check which version of the XTNetFile client we >>> >>>are using? Is >>> >>>>it part of the release or a shared library installed >>> >>>somewhere else? How >>> >>>>can we update? >>>> >>>>Hopefully with Manny's fix to the staging system we won't see this >>>>problem so often, but from what Andy says it's probably >>> >>>still lurking. >>> >>>>Thanks, >>>>Tim. >>>> >>>> >>>>>-----Original Message----- >>>>>From: Andrew Hanushevsky <[log in to unmask]> >>>>>Sent: 09 June 2005 23:36 >>>>>To: Bill Weeks >>>>>Cc: [log in to unmask]; Adye, TJ (Tim); Brew, CAJ (Chris); >>>>>Fabrizio Furano; [log in to unmask] >>>>>Subject: Re: PreStage Problems >>>>> >>>>>Hi Manny, >>>>> >>>>>I guess we will completely sort this out on Monday. >>>>>Distilling all of the below, there are only one saliant issue: >>>>> >>>>>a) Why is the file *not* getting basedir prepended to it? We >>>>>can figure this out by doing a diff on what you installed and >>>>>what is in utils to see why mps_PreStage is not prefixing >>> >>>the path. >>> >>>>>The "continuing to hang" problem is a client problem. Here >>>>>the client is always asking for a cache refresh. So, either >>>>>an old client is being used (old clients had this bug and it >>>>>was fixed about 6 months ago) or the bug has returned under >>>>>this new scenario (I suspect that the latter is true). >>>>> >>>>>So, Fabrizio, do you see anywhere in the client where the >>>>>code may get causght in a cache refresh loop? >>>>>Andy >>>>> >>>>>On Thu, 9 Jun 2005, Bill Weeks wrote: >>>>> >>>>> >>>>>>Hi, >>>>>>I hope I can help sort out what's going on here, but it is >>>>> >>>>>confusing. >>>>> >>>>>>First off, mps_PreStage and mps_Stage never really >>> >>>handled "mssdir" >>> >>>>>>and "basedir" correctly. This was never a problem for us >>>>> >>>>>because these >>>>> >>>>>>have always been the same. For RAL, this is not the case. So RAL >>>>>>(Chris?) changed mps_PreStage to add $basedir to the target >>>>> >>>>>filename, e.g. >>>>> >>>>>> $cmd = "$pstgcmd $rflag $Lflag $file $basedir/$file 2>&1"; >>>>>> >>>>>>Once this was done, mps_Stage failed for a file whose >>> >>>path did not >>> >>>>>>previously exist because $basedir/$file created a filepath >>>>> >>>>>with a "//" >>>>> >>>>>>in it and the MakePath subroutine didn't handle this >>> >>>properly. The >>> >>>>>>change I made in version 1.9 of mps_Stage removed the >>>>> >>>>>double //'s so >>>>> >>>>>>MakePath would work properly. >>>>>> >>>>>>The problem you are now reporting seems to indicate >>> >>>that you have >>> >>>>>>either removed your mod to mps_PreStage or have redefined >>>>> >>>>>basedir in >>>>> >>>>>>your config file because mps_Stage is trying to write >>> >>>into /store >>> >>>>>>instead of /basedir/store, e.g. >>>>> >>>>>/stage/bdata-data50/kanga/store. Is this what happened? >>>>> >>>>>>I think once the file is correctly staged in, the >>> >>>waiting jobs that >>> >>>>>>are polling for the file will continue. >>>>>> >>>>>>We still have some work to do to correctly handle the >>>>> >>>>>situation where >>>>> >>>>>>mssdir and basedir are different. >>>>>>--Bill Weeks, SLAC, (650) 926-2909 >>>>>> >>>>>> >>>>>> >>>>>>>Date: Tue, 07 Jun 2005 14:30:56 -0700 >>>>>>>From: Emmanuel Olaiya <[log in to unmask]> >>>>>>>User-Agent: Mozilla Thunderbird 0.9 (X11/20041103) >>>>>>>X-Accept-Language: en-us, en >>>>>>>MIME-Version: 1.0 >>>>>>>To: Andrew Hanushevsky <[log in to unmask]> >>>>>>>CC: "Adye, TJ (Tim)" <[log in to unmask]>, "Brew, CAJ (Chris)" >>>>>> >>>>>><[log in to unmask]>, [log in to unmask], Bill Weeks >>>>>><[log in to unmask]> >>>>>> >>>>>>>Subject: Re: PreStage Problems >>>>>>>Content-Transfer-Encoding: 7bit >>>>>>> >>>>>>>Hi Andy, Bill >>>>>>> >>>>>>>I took the versions of mps_Stage and mps_prep from >>>>>>>/afs/slac/package/xrd/xrootd/utils. These are mps_Stage >>>>> >>>>>and mps_prep >>>>> >>>>>>>versions 1.9 and 1.8 respectively. >>>>>>> >>>>>>>I still see the problem Chris reported. Restarting the >>>>> >>>>>directors and >>>>> >>>>>>>the server (with prestaging on the server) I get the following >>>>>>>message in the prestage log when asking for a file that >>>>> >>>>>doesn't exist >>>>> >>>>>>>at RAL >>>>>>> >>>>>>>Starting new cycle, pstg proc = 0 >>>>>>>21:17:41 [ 17543] getlock: locking file >>>>>>> >>>>>>>>>/opt/xrootd/stageQ/PreStageQ.0.lock, flags 2 >>>>>>> >>>>>>>21:17:41 [ 17543] getlock: locking file >>>>>>>+</opt/xrootd/stageQ/PreStageQ.0.old, flags 2 >>>>>>>21:17:41 [ 17543] unlock: unlocking file >>>>>>>/opt/xrootd/stageQ/PreStageQ.0.old >>>>>>>21:17:41 [ 17543] unlock: unlocking file >>>>>>>/opt/xrootd/stageQ/PreStageQ.0.lock >>>>>>>21:17:41 [ 17543] getlock: locking file >>>>>>> >>>>>>>>>/opt/xrootd/stageQ/PreStageQ.1.lock, flags 2 >>>>>>> >>>>>>>21:17:41 [ 17543] unlock: unlocking file >>>>>>>/opt/xrootd/stageQ/PreStageQ.1.lock >>>>>>>21:21:29 [ 17772] mps_Stage: cannot create 'store' in >>>>>>>'/store/PRskims/R14/16.1.1b/BToPPP/58/'; Permission denied >>>>>>>21:21:29 [ 17772] mps_Stage: Invalid file system path, >>>>>>>'/store/PRskims/R14/16.1.1b/BToPPP/58/'. >>>>>>>21:21:29 [ 17772] do_stagein: xfr failed for >>>>>>>/store/PRskims/R14/16.1.1b/BToPPP/58/BToPPP_5831.01.root, rc=4, >>>>>>>retry=1 >>>>>>> >>>>>>>Whilst my job just hangs. If I take the log file >>> >>>literally, it is >>> >>>>>>>trying to write to /store when it should be trying to write to >>>>>>>/base_directory/store. >>>>>>> >>>>>>>Doing further tests I can reproduce the problem I >>> >>>reported earlier. >>> >>>>>>>Whilst still asking for the above file I turn off >>> >>>staging, restart >>> >>>>>>>the directors and servers and the request for the file >>>>> >>>>>continues to >>>>> >>>>>>>hang (is told to wait). Then I make another request for >>>>> >>>>>the same file >>>>> >>>>>>>and this request is also continually told to wait: >>>>>>> >>>>>>>050607 21:55:13 2915 odc_Locate: >>>>> >>>>>olaiya.8042:[log in to unmask] asked >>>>> >>>>>>>to wait 5 by xrootd107 >>>>>>>path=/store/PRskims/R14/16.1.1b/BToPPP/58/BToPPP_5831.01.root >>>>>>>050607 21:55:14 2915 odc_Locate: >>>>> >>>>>olaiya.23507:[log in to unmask] asked >>>>> >>>>>>>to wait 5 by xrootd107 >>>>>>>path=/store/PRskims/R14/16.1.1b/BToPPP/58/BToPPP_5831.01.root >>>>>>>050607 21:55:18 2915 odc_Locate: >>>>> >>>>>olaiya.8042:[log in to unmask] asked >>>>> >>>>>>>to wait 5 by xrootd107 >>>>>>>path=/store/PRskims/R14/16.1.1b/BToPPP/58/BToPPP_5831.01.root >>>>>>>... >>>>>>> >>>>>>> >>>>>>>It is only after I kill the first request that anymore >>>>> >>>>>requests for >>>>> >>>>>>>this file return correctly with a message indicating >>> >>>that the file >>> >>>>>>>cannot be found. >>>>>>> >>>>>>>cheers >>>>>>> >>>>>>>Manny >>>>>>> >>>>>>>Andrew Hanushevsky wrote: >>>>>>> >>>>>>>>Hi Tim, >>>>>>>> >>>>>>>>Bill Weeks should have the fix available. You can >>> >>>also find the >>> >>>>>>>>fixed mps scripts in >>> >>>/afs/slac/package/xrd/xrootd/utils (I think >>> >>>>>>>>you just need an update for mps_Stage and mps_prep). >>>>>>>> >>>>>>>>Otherwise, the earliest time I can get together with Many is >>>>>>>>Monday. How about the afternoon, say 1:30pm? >>>>>>>> >>>>>>>>Andy >>>>>>>> >>>>>>>>On Tue, 7 Jun 2005, Adye, TJ (Tim) wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Hi Guys, >>>>>>>>> >>>>>>>>>Did you manage to sort something out, despite the >>>>> >>>>>cancellation of >>>>> >>>>>>>>>the meeting? These are serious problems for us. >>>>>>>>> >>>>>>>>>Tim. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>-----Original Message----- >>>>>>>>>>From: [log in to unmask] >>>>>>>>>>[mailto:[log in to unmask]] On >>> >>>Behalf Of >>> >>>>>>>>>>Emmanuel Olaiya >>>>>>>>>>Sent: 06 June 2005 22:57 >>>>>>>>>>To: Andy Hanushevsky >>>>>>>>>>Cc: Brew, CAJ (Chris); [log in to unmask]; >>> >>>Bill Weeks >>> >>>>>>>>>>Subject: Re: PreStage Problems >>>>>>>>>> >>>>>>>>>>Hi Andy >>>>>>>>>> >>>>>>>>>>Yes, it would be good if you could have a look at this >>>>> >>>>>with me. We >>>>> >>>>>>>>>>can arrange a time in the xrootd meeting tomorrow. >>>>>>>>>> >>>>>>>>>>cheers >>>>>>>>>> >>>>>>>>>>Manny >>>>>>>>>> >>>>>>>>>>Andy Hanushevsky wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>Hi Manny, >>>>>>>>>>> >>>>>>>>>>>I find this is quite mysterious as this should never be the >>>>>>>>>> >>>>>>>>>>case and, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>frankly, appears to violate causality. I suspect something >>>>>>>>>> >>>>>>>>>>else is going >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>on. If this is reproducible then why don't we run a >>>>> >>>>>test with all >>>>> >>>>>>>>>>>debugging turned on. Yes? >>>>>>>>>>> >>>>>>>>>>>Andy >>>>>>>>>>> >>>>>>>>>>>----- Original Message ----- From: "Emmanuel Olaiya" >>>>>>>>>> >>>>>>>>>><[log in to unmask]> >>>>>>>>>> >>>>>>>>>>>To: "Andrew Hanushevsky" <[log in to unmask]> >>>>>>>>>>>Cc: "Brew, CAJ (Chris)" <[log in to unmask]>; >>>>>>>>>>><[log in to unmask]>; "Bill Weeks" >>>>>>>>>>><[log in to unmask]> >>>>>>>>>>>Sent: Monday, June 06, 2005 1:41 PM >>>>>>>>>>>Subject: Re: PreStage Problems >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>Hi Andy >>>>>>>>>>>> >>>>>>>>>>>>I should have mentioned that we also remove the >>>>> >>>>>prestage queue >>>>> >>>>>>>>>>>>and restarted both the server and redirector. >>> >>>However the old >>> >>>>>>>>>>request to >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>wait did not change. Moreover, any similar new requests >>>>>>>>>> >>>>>>>>>>were also told >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>to wait until the old request was terminated. >>>>>>>>>>>> >>>>>>>>>>>>cheers >>>>>>>>>>>> >>>>>>>>>>>>Manny >>>>>>>>>>>> >>>>>>>>>>>>Andrew Hanushevsky wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>Hi Manny, >>>>>>>>>>>>> >>>>>>>>>>>>>Yes, but who telling the client to wait? The redirector >>>>>>>>>> >>>>>>>>>>or the server >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>that >>>>>>>>>>>>>wanted to orginally stage the file in. When you >>> >>>restart the >>> >>>>>>>>>>>>>redirector it loses all it's memory but the data >>> >>>server does >>> >>>>>>>>>>>>>not. So, >>>>>>>>>> >>>>>>>>>>it will hapiily >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>tell the redirector that it has the file eventhough >>>>> >>>>>the file is >>>>> >>>>>>>>>>>>>merely in the pre-stage queue. As long as the >>> >>>file is in the >>> >>>>>>>>>>prestage queue and >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>not on >>>>>>>>>>>>>disk, the only option is to direct clients to where the >>>>>>>>>> >>>>>>>>>>file will be >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>staged in and then the clients simply wait for the file >>>>>>>>>> >>>>>>>>>>(which in this >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>case will never appear). So, if you remove staging you >>>>>>>>>> >>>>>>>>>>also need to >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>remove >>>>>>>>>>>>>the prestage queue and restart the data server. >>>>>>>>>>>>> >>>>>>>>>>>>>Andy >>>>>>>>>>>>> >>>>>>>>>>>>>On Fri, 3 Jun 2005, Emmanuel Olaiya wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>Hi Andy >>>>>>>>>>>>>> >>>>>>>>>>>>>>One other issue we have spotted at RAL. We removed >>>>> >>>>>the staging >>>>> >>>>>>>>>>>>>>capabilities and restarted the director and server. >>>>>>>>>> >>>>>>>>>>However we found >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>previous requests for a file that were told to wait >>>>>>>>>> >>>>>>>>>>continued being >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>told >>>>>>>>>>>>>>to wait. We also found that if somebody else asked for >>>>>>>>>> >>>>>>>>>>this same file >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>that was not on disk they were also told to wait rather >>>>>>>>>> >>>>>>>>>>than being told >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>the file could not be found. We needed to kill the >>>>>>>>>> >>>>>>>>>>previous request and >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>restart the server and directory for xrootd to know the >>>>>>>>>> >>>>>>>>>>file was not on >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>disk. >>>>>>>>>>>>>> >>>>>>>>>>>>>>cheers >>>>>>>>>>>>>> >>>>>>>>>>>>>>Manny >>>>>>>>>>>>>> >>>>>>>>>>>>>>Andrew Hanushevsky wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>Hi Chris, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Oh yeah, different problem. I think that Bill >>>>> >>>>>Weeks fixed that. >>>>> >>>>>>>>>>>>>>>Bill did >>>>>>>>>>>>>>>you fix that problem? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Andy >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>On Mon, 30 May 2005, Brew, CAJ (Chris) wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I might be being stupid but I don't see how this >>>>>>>>>> >>>>>>>>>>relates to the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>problem. >>>>>>>>>>>>>>>>The files I wanted were on a different disk server >>>>>>>>>> >>>>>>>>>>which then went >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>down. >>>>>>>>>>>>>>>>The server in question was registered with the OLB as >>>>>>>>>> >>>>>>>>>>being able to >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>stage in the name space so the request was >>>>> >>>>>redirected to it. >>>>> >>>>>>>>>>>>>>>>If mps_Stage is used without the PreStage >>> >>>queuing system >>> >>>>>>>>>>everything >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>works >>>>>>>>>>>>>>>>as expected. If we try to go through the PreStage >>>>>>>>>> >>>>>>>>>>queue to limit the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>number of concurrent accesses to the tapestore the >>>>>>>>>> >>>>>>>>>>stage in fails. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>Apparently because the DIR_LOCK file does not >>>>> >>>>>exist (which >>>>> >>>>>>>>>>>>>>>>it doesn't, since the file, and it's >>> >>>directory structure, >>> >>>>>>>>>>>>>>>>has >>>>>>>>>> >>>>>>>>>>never existed on >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>this >>>>>>>>>>>>>>>>server). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Yours, >>>>>>>>>>>>>>>>Chris. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>From: Andrew Hanushevsky >>> >>>[mailto:[log in to unmask]] >>> >>>>>>>>>>>>>>>>>Sent: 28 May 2005 07:39 >>>>>>>>>>>>>>>>>To: Brew, CAJ (Chris) >>>>>>>>>>>>>>>>>Cc: [log in to unmask]; abh; Olaiya, EO >>>>> >>>>>(Emmanuel) >>>>> >>>>>>>>>>>>>>>>>Subject: RE: PreStage Problems >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Hi Chris, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>This was traced to overzealous testing. The >>> >>>syustem does >>> >>>>>>>>>>>>>>>>>not put in a new entry in the pre-stage queue >>>>> >>>>>until after >>>>> >>>>>>>>>>>>>>>>>about 10-20 minutes have elapsed since the >>> >>>last time the >>> >>>>>>>>>>>>>>>>>entry was added. So, this is not a bug but a >>>>> >>>>>test case that >>>>> >>>>>>>>>>>>>>>>>was not "real". Generally, files live in the >>>>> >>>>>disk cache for >>>>> >>>>>>>>>>>>>>>>>at least 10-20 minutes. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Andy >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>On Fri, 27 May 2005, Brew, CAJ (Chris) wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>At the meeting a couple of weeks ago, it was said >>>>>>>>>> >>>>>>>>>>that someone was >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>looking into this but I haven't heard >>> >>>anything back. Is >>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>there any new? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>>Chris. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>>>From: Brew, CAJ (Chris) >>>>>>>>>>>>>>>>>>>Sent: 17 May 2005 13:50 >>>>>>>>>>>>>>>>>>>To: [log in to unmask]; abh >>>>>>>>>>>>>>>>>>>Cc: Olaiya, EO (Emmanuel) >>>>>>>>>>>>>>>>>>>Subject: PreStage Problems >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>I've been running some more tests of the >>>>> >>>>>staging at RAL >>>>> >>>>>>>>>>>>>>>>>>>and have run into a problem somewhere in the >>>>>>>>>>>>>>>>>>>mps_Stage/PreStage/prep system. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>Everything work fine staging file that was on >>>>> >>>>>the system >>>>> >>>>>>>>>>>>>>>>>>>and has been deleted but if I try to stage >>> >>>in a file >>> >>>>>>>>>>that was one >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>a different server, hence the directory >>>>> >>>>>structure for the >>>>> >>>>>>>>>>>>>>>>>>>file does not exist on the staging server it >>>>> >>>>>fails and I >>>>> >>>>>>>>>>>>>>>>>>>see the following error in the PreStage log file: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>12:45:43 [ 10859] mps_Stage: Open >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>'/stage/bdata-data50/kanga//store/SPskims/R12/16.0.2e/BtoKKKL/ >>>>>>> >>>>>>>>>>>>>>>>>>>001005/200002/DIR_LOCK' r/w failed; No such file or >>>>>>>>>> >>>>>>>>>>directory. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>>12:45:43 [ 10859] do_stagein: xfr failed for >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>/store/SPskims/R12/16.0.2e/BtoKKKL/001005/200002/BtoKKKL_00100 >>>>>>> >>>>>>>>>>>>>>>>>>>5_3247.01.root, rc=4, retry=1 >>>>>>>>>>>>>>>>>>>12:45:45 [ 3255] >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>file=/store/SPskims/R12/16.0.2e/BtoKKKL/001005/200002/BtoKKKL_ >>>>>>> >>>>>>>>>>>>>>>>>>>0010053247.01.root, rc=1024, >>>>> >>>>>reqid=ef000001:1cd2.425d27e1 >>>>> >>>>>>>>>>>>>>>>>>>:3762 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>If I create the directories and the DIR_LOCK >>>>> >>>>>file before >>>>> >>>>>>>>>>>>>>>>>>>running the import, everything works. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>The config file I'm using on the server is below. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>Is there some setting I'm missing which is >>> >>>needed to >>> >>>>>>>>>>>>>>>>>>>create the directories/DIR_LOCK file or does >>>>> >>>>>the code need fixing? >>>>> >>>>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>>>Chris >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>-- >>>>>>>>>>>>>>>>>>>Chris Brew ([log in to unmask]) +44 1235 446326 >>>>>>>>>>>>>>>>>>>Particle Physics Department Rutherford Appleton >>>>>>>>>>>>>>>>>>>Laboratory Chilton, Didcot. Oxfordshire. >>>>>>>>>>>>>>>>>>>OX11 0QX. United Kingdom. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>-- >>> /------------------------------------+-------------------------\ >>>|Stephen J. Gowdy | SLAC, MailStop 34, | >>>|http://www.slac.stanford.edu/~gowdy/ | 2575 Sand Hill Road, | >>>|http://calendar.yahoo.com/gowdy | Menlo Park CA 94025, USA | >>>|EMail: [log in to unmask] | Tel: +1 650 926 3144 | >>> \------------------------------------+-------------------------/ >>> >> > > -- > /------------------------------------+-------------------------\ > |Stephen J. Gowdy | SLAC, MailStop 34, | > |http://www.slac.stanford.edu/~gowdy/ | 2575 Sand Hill Road, | > |http://calendar.yahoo.com/gowdy | Menlo Park CA 94025, USA | > |EMail: [log in to unmask] | Tel: +1 650 926 3144 | > \------------------------------------+-------------------------/