URL:
<http://savannah.cern.ch/support/?136498>
Summary: Problems accessing files when using "?tried=..."
Project: XROOTD
Submitted by: alahiff
Submitted on: 2013-03-14 15:06
Category: None
Priority: 5 - Normal
Severity: 3 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Operating System: GNU/Linux
_______________________________________________________
Details:
Hi,
At RAL we are providing xrootd access to CMS files which are on disk in
CASTOR. Some remote sites append something like
"?tried=gfe02.grid.hep.ph.ic.ac.uk", for example, in their xrootd fallback
configuration. However, this causes the following error to occur:
"Unable to reqid/pfn2 is missing dummy; Cannot send after transport
endpoint shutdown(error code: 3005)"
See below [1] for a full example.
This can easily be reproduced, e.g.
[lxplus435] ~ $ xrdcp
root://cms-xrd-global.cern.ch//store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root\?tried=gfe02.grid.hep.ph.ic.ac.uk
/var/tmp/test1
Last server error 3005 ('Unable to reqid/pfn2 is missing dummy; Cannot send
after transport endpoint shutdown')
Error accessing path/file for
root://cms-xrd-global.cern.ch//store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root?tried=gfe02.grid.hep.ph.ic.ac.uk
and in /var/log/xroot/server/xrdlog.server on the appropriate diskserver is:
130314 14:55:32 4667 XrootdXeq: alahiff.5522:[log in to unmask] login as
alahiff
130314 14:55:32 4667 ofs_open: alahiff.5522:[log in to unmask] Unable to
reqid/pfn2 is missing dummy; Cannot send after transport endpoint shutdown
130314 14:55:32 4667 XrootdXeq: alahiff.5522:[log in to unmask] disc
0:00:00
Does anyone know what could be causing this? Is there a workaround that we
could use, e.g. stripping out the "?tried=..." somehow?
We have the following xrootd rpms installed on our diskservers:
xrootd-xcastor2fs_2112-debuginfo-1.1.0-1
xrootd-server-3.0.4-0
xrootd-devel-3.0.4-0
xrootd-xcastor2fs_2112-1.1.0-1
The error seems to be generated by the diskserver; the xrootd manager doesn't
give any errors in its logs [2].
Please let me know if there's any more information required.
Many Thanks,
Andrew.
[1]
08-Mar-2013 16:04:29 GMT Initiating request to open
fileroot://gfe02.grid.hep.ph.ic.ac.uk:1097//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root
130308 16:04:29 16998 Xrd: CheckErrorStatus: Server
[gfe02.grid.hep.ph.ic.ac.uk:1097] declared: No such file(error code: 3004)
08-Mar-2013 16:04:29 GMT Fallback request to file
root://cms-xrd-global.cern.ch,xrootd.ba.infn.it,xrootd.unl.edu//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root?tried=gfe02.grid.hep.ph.ic.ac.uk
130308 16:04:29 16998 Xrd: GoToAnotherServer: Going to:
xrootd.ba.infn.it:1094
130308 16:04:30 16998 Xrd: GoToAnotherServer: Going to:
lcgclsf02.gridpp.rl.ac.uk:1094
130308 16:04:30 16998 Xrd: GoToAnotherServer: Going to:
lcgcdlf02.gridpp.rl.ac.uk:1094
130308 16:04:30 16998 Xrd: GoToAnotherServer: Going to:
gdss517.gridpp.rl.ac.uk:1095
130308 16:04:30 16998 Xrd: CheckErrorStatus: Server
[lcgcdlf02.gridpp.rl.ac.uk:1094] declared: Unable to reqid/pfn2 is missing
dummy; Cannot send after transport endpoint shutdown(error code: 3005)
----- Begin Fatal Exception 08-Mar-2013 16:04:30 GMT-----------------------
An exception of category 'FallbackFileOpenError' occurred while
[0] Constructing the EventProcessor
[1] Constructing input source of type PoolSource
[2] Calling RootInputFileSequence::initFile()
[3] Calling StorageFactory::open()
[4] Calling XrdFile::open()
Exception Message:
Input file
root://gfe02.grid.hep.ph.ic.ac.uk:1097//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root
was not found, could not be opened, or is corrupted.
Fallback Input file
root://cms-xrd-global.cern.ch,xrootd.ba.infn.it,xrootd.unl.edu//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root?tried=gfe02.grid.hep.ph.ic.ac.uk
also was not found, could not be opened, or is corrupted.
Additional Info:
[a]
XrdClient::Open(name='root://cms-xrd-global.cern.ch,xrootd.ba.infn.it,xrootd.unl.edu//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root?tried=gfe02.grid.hep.ph.ic.ac.uk',
flags=0x10, permissions=0666) => error 'Unable to reqid/pfn2 is missing dummy;
Cannot send after transport endpoint shutdown' (errno=3005)
[b] Current server connection:
root://gdss517.gridpp.rl.ac.uk:1095//store/mc/Summer12_DR53X/RPVsmuon_M0-1000_M12-200_TuneZ2star_8TeV-calchep-pythia6-tauola/AODSIM/PU_S10_START53_V7C-v1/20000/22877F4E-D682-E211-A0F9-003048C68A92.root?tried=gfe02.grid.hep.ph.ic.ac.uk
[2]
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_stat:
fn=/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_stat: getgid
202/202
fn=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_GetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
Opaque=oss.lcl=1
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_SetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
Opaque=oss.lcl=1 Stagevariables=ccmsstager.ads.rl.ac.uk::cmsTape
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_SetStageVariables:
0
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost= dserviceclass= stagehost=ccmsstager.ads.rl.ac.uk
serviceclass=cmsTape n=0 ntoken=0
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_SetStageVariables:
1
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost=ccmsstager.ads.rl.ac.uk dserviceclass=
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape n=0 ntoken=0
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_SetStageVariables:
2
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost=ccmsstager.ads.rl.ac.uk dserviceclass=cmsTape
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape n=0 ntoken=0
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_SetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_StagerQuery:
uid=202 gid=202
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_StagerQuery:
status=STAGED rc=0
path=gdss528.gridpp.rl.ac.uk:[log in to unmask]
(/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root)
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_StagerQuery: File
status: STAGED for path:
/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:30 4046 xrootd.20872:211@lcgclsf02 xcastor2fs_stat: STAGED
fn=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_open:
fn=/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_open: 0-600
fn =
/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_open:
getgid 36782/24327
fn=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_GetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
Opaque=
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_SetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
Opaque= Stagevariables=ccmsstager.ads.rl.ac.uk::cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_SetStageVariables: 0
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost= dserviceclass= stagehost=ccmsstager.ads.rl.ac.uk
serviceclass=cmsTape n=0 ntoken=0
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_SetStageVariables: 1
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost=ccmsstager.ads.rl.ac.uk dserviceclass=
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape n=0 ntoken=0
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_SetStageVariables: 2
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
dstagehost=ccmsstager.ads.rl.ac.uk dserviceclass=cmsTape
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape n=0 ntoken=0
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_SetStageVariables:
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_StagerQuery: uid=36782 gid=24327
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_StagerQuery: status=STAGED rc=0
path=gdss528.gridpp.rl.ac.uk:[log in to unmask]
(/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root)
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_StagerQuery: File status: STAGED for path:
/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_Prepare2Get: uid=36782 gid=24327
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_Prepare2Get: Sending Prepare2Get
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
uid=36782 gid=24327 stagehost=ccmsstager.ads.rl.ac.uk svc=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask]
xcastor2fs_Prepare2Get: READYstatus=6 rc=0
path=gdss528.gridpp.rl.ac.uk:[log in to unmask]
(/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root)
id=4674364954
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_Get:
uid=36782 gid=24327
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
stagehost=ccmsstager.ads.rl.ac.uk serviceclass=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_Get:
Sending Get
path=/castor/ads.rl.ac.uk/prod/cms/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root
uid=36782 gid=24327 stagehost=ccmsstager.ads.rl.ac.uk svc=cmsTape
130314 14:55:32 4046 alahiff.5522:[log in to unmask] xcastor2fs_open:
redirection to
gdss528.gridpp.rl.ac.uk?castor2fs.sfn=/store/data/Run2012D/TauPlusX/RAW-RECO/MuTauMET-22Jan2013-v1/20000/EE1AF7DE-BC89-E211-8B07-00266CFFA750.root&[log in to unmask]&castor2fs.pfn2=4674364960:ccmsstager.ads.rl.ac.uk:cmsTape:63632:d7bed6f4-0efd-206a-e043-17b6f682122f&castor2fs.id=alahiff.5522:[log in to unmask]&castor2fs.client_sec_uid=36782&castor2fs.client_sec_gid=24327&castor2fs.accessop=8&castor2fs.exptime=1363272992&[log in to unmask]:ccmsstager.ads.rl.ac.uk:cmsTape:63632:d7bed6f4-0efd-206a-e043-17b6f682122falahiff.5522:[log in to unmask]:1094&castor2fs.signature=tcEZ8Dj8asLIcJqU3hYUKHT67rqGniEz/RMyh8GVSFBaoOWfe/aJgwoWIFEzL3idkK6xKbF+DLcXwAX20fFoQg==&castor2fs.manager=lcgcdlf02.gridpp.rl.ac.uk:1094&adler32=6914d6a&
_______________________________________________________
Reply to this item at:
<http://savannah.cern.ch/support/?136498>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
