URL:
<http://savannah.cern.ch/bugs/?91129>
Summary: XrdClient deadlocks on auth failure
Project: XROOTD
Submitted by: bbockelm
Submitted on: 2012-02-03 01:27
Severity: 3 - Normal
Priority: 5 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Fixed by commit(s):
_______________________________________________________
Details:
Hi,
I think, in the stack trace below, Thread #1 has caused a deadlock by
disconnecting after an authentication failure. This happens 100% of the time
in my current tests. Should be Xrootd 3.1.0 (as shipped with CMSSW).
I've also copied the client session at debug level 1.
Brian
[bbockelm@t3 CMSSW_5_0_1]$ pstack 7484
Thread 3 (Thread 0x4256a940 (LWP 7485)):
#0 0x000000392240d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003922408e35 in _L_lock_1127 () from /lib64/libpthread.so.0
#2 0x0000003922408d33 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b6f423f8480 in XrdClientConnectionMgr::GarbageCollect() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#4 0x00002b6f423f8745 in GarbageCollectorThread(void*, XrdClientThread*) ()
from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#5 0x00002b6f42436d37 in XrdSysThread_Xeq () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#6 0x000000392240673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003921cd44bd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4396c940 (LWP 7488)):
#0 0x000000392240d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003922408e35 in _L_lock_1127 () from /lib64/libpthread.so.0
#2 0x0000003922408d33 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b6f424019b2 in XrdClientPhyConnection::Disconnect() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#4 0x00002b6f42402722 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#5 0x00002b6f423ffc0c in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#6 0x00002b6f42401f0a in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#7 0x00002b6f4240362a in SocketReaderThread(void*, XrdClientThread*) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#8 0x00002b6f42436d37 in XrdSysThread_Xeq () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#9 0x000000392240673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003921cd44bd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b6f410d8f70 (LWP 7484)):
#0 0x000000392240d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003922408e35 in _L_lock_1127 () from /lib64/libpthread.so.0
#2 0x0000003922408d33 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b6f42400f70 in XrdClientPhyConnection::IsValid() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#4 0x00002b6f423f81fb in DumpPhyConn(char const*, XrdClientPhyConnection*,
void*) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#5 0x00002b6f423fba1e in XrdOucHash<XrdClientPhyConnection>::Apply(int
(*)(char const*, XrdClientPhyConnection*, void*), void*) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#6 0x00002b6f423f84d1 in XrdClientConnectionMgr::GarbageCollect() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#7 0x00002b6f423fa09c in XrdClientConnectionMgr::Disconnect(int, bool) ()
from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#8 0x00002b6f423e83e0 in XrdClientConn::Disconnect(bool) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#9 0x00002b6f423f4dfd in XrdClientConn::DoLogin() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#10 0x00002b6f423f5c78 in XrdClientConn::GetAccessToSrv() () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#11 0x00002b6f423dd153 in XrdClient::Open(unsigned short, unsigned short,
bool) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libXrdClient.so
#12 0x00002b6f423b9e96 in XrdFile::open(char const*, int, int) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libUtilitiesXrdAdaptor.so
#13 0x00002b6f423ba61d in XrdFile::XrdFile(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, int, int) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libUtilitiesXrdAdaptor.so
#14 0x00002b6f423b2031 in XrdStorageMaker::open(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, int) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginUtilitiesXrdAdaptorPlugin.so
#15 0x00002b6f4172a3e1 in StorageFactory::open(std::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, int) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libUtilitiesStorageFactory.so
#16 0x00002b6f416c9a64 in TStorageFactoryFile::TStorageFactoryFile(char
const*, char const*, char const*, int, int, bool) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolTFileAdaptor.so
#17 0x00002b6f416d066f in G__TFileAdaptorLinkDef_220_0_13(G__value*, char
const*, G__param*, int) () from
/home/cse496/bbockelm/tmp/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolTFileAdaptor.so
#18 0x00002b6f402de5dc in Cint::G__CallFunc::Execute(void*) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libCint.so
#19 0x00002b6f3fd089d1 in TCint::CallFunc_ExecInt(void*, void*) const () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libCore.so
#20 0x00002b6f3fd36634 in TMethodCall::Execute(void*, long&) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libCore.so
#21 0x00002b6f3fc98b14 in TPluginHandler::ExecPlugin(int, ...) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libCore.so
#22 0x00002b6f3f83bbb6 in TFile::Open(char const*, char const*, char const*,
int, int) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/external/slc5_amd64_gcc434/lib/libRIO.so
#23 0x00002b6f421d33f9 in edm::InputFile::InputFile(char const*, char const*)
() from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#24 0x00002b6f42204043 in edm::RootInputFileSequence::initFile(bool) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#25 0x00002b6f4220887f in
edm::RootInputFileSequence::RootInputFileSequence(edm::ParameterSet const&,
edm::PoolSource const&, edm::InputFileCatalog const&, edm::PrincipalCache&,
edm::InputType::InputType) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#26 0x00002b6f421d8b11 in edm::PoolSource::PoolSource(edm::ParameterSet
const&, edm::InputSourceDescription const&) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#27 0x00002b6f421d5434 in edmplugin::PluginFactory<edm::InputSource*
()(edm::ParameterSet const&, edm::InputSourceDescription
const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&,
edm::InputSourceDescription const&) const () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#28 0x00002b6f3e1f1495 in
edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&,
edm::InputSourceDescription const&) const () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#29 0x00002b6f3e1925fb in edm::makeInput(edm::ParameterSet&,
edm::CommonParams const&, edm::ProductRegistry&, edm::PrincipalCache&,
boost::shared_ptr<edm::ActivityRegistry>,
boost::shared_ptr<edm::ProcessConfiguration>) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#30 0x00002b6f3e194a9e in
edm::EventProcessor::init(boost::shared_ptr<edm::ProcessDesc>&,
edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#31 0x00002b6f3e196c9d in
edm::EventProcessor::EventProcessor(boost::shared_ptr<edm::ProcessDesc>&,
edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) () from
/opt/osg/app/cmssoft/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_5_0_1/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#32 0x000000000040f0a4 in main ()
[bbockelm@t3 CMSSW_5_0_1]$ cmsRun simple.py -j test2.fjr
120202 19:18:16 001 Xrd: CheckErrorStatus: Server [brian-test.unl.edu]
declared: (error code: 3010)
120202 19:18:16 001 Xrd: DoAuthentication:
XrdSec: No authentication protocols are available.
120202 19:18:16 001 Xrd: Connect: Authentication failure: : unable to get
protocol object
02-Feb-2012 19:18:16 CST Initiating request to open file
root://brian-test.unl.edu//store/relval/CMSSW_5_0_1/Mu/RECO/GR_R_50_V11_RelVal_mu2010B-v1/0224/6289ACC2-4E4B-E111-B830-002618943971.root
120202 19:18:16 001 Xrd: ShowUrls: The converted URLs count is 1
120202 19:18:16 001 Xrd: ShowUrls: URL n.1:
root://brian-test.unl.edu:1094//store/relval/CMSSW_5_0_1/Mu/RECO/GR_R_50_V11_RelVal_mu2010B-v1/0224/6289ACC2-4E4B-E111-B830-002618943971.root.
120202 19:18:16 001 Xrd: ShowUrls: The converted URLs count is 1
120202 19:18:16 001 Xrd: ShowUrls: URL n.1:
root://brian-test.unl.edu:1094//store/relval/CMSSW_5_0_1/Mu/RECO/GR_R_50_V11_RelVal_mu2010B-v1/0224/6289ACC2-4E4B-E111-B830-002618943971.root.
120202 19:18:16 001 Xrd: CheckErrorStatus: Server [brian-test.unl.edu]
declared: (error code: 3010)
120202 19:18:16 001 Xrd: DoAuthentication:
XrdSec: No authentication protocols are available.
120202 19:18:16 001 Xrd: CheckErrorStatus: Server [brian-test.unl.edu]
declared: Invalid request; user not authenticated(error code: 3006)
120202 19:18:16 001 Xrd: DumpPhyConn: Phyconn entry,
[log in to unmask]:1094', LogCnt=1 NotValid
120202 19:18:16 001 Xrd: DumpPhyConn: Phyconn entry, key='Trashed
connection', LogCnt=1 NotValid
120202 19:18:16 001 Xrd: DumpPhyConn: Phyconn entry, key='Trashed
connection', LogCnt=0 NotValid
120202 19:18:16 001 Xrd: Open: Access to server failed: error: 3006 (Invalid
request; user not authenticated) - retrying.
120202 19:18:16 001 Xrd: Open: Connection attempt failed. Sleeping 5
seconds.
120202 19:18:16 7484 Xrd: XrdClientMessage::ReadRaw: Failed to read header (8
bytes).
120202 19:18:17 7484 Xrd: XrdClientMessage::ReadRaw: Failed to read header (8
bytes).
120202 19:18:22 001 Xrd: ShowUrls: The converted URLs count is 1
120202 19:18:22 001 Xrd: ShowUrls: URL n.1:
root://brian-test.unl.edu:1094//store/relval/CMSSW_5_0_1/Mu/RECO/GR_R_50_V11_RelVal_mu2010B-v1/0224/6289ACC2-4E4B-E111-B830-002618943971.root.
120202 19:18:22 001 Xrd: CheckErrorStatus: Server [brian-test.unl.edu]
declared: (error code: 3010)
120202 19:18:22 001 Xrd: DoAuthentication:
XrdSec: No authentication protocols are available.
120202 19:18:22 001 Xrd: CheckErrorStatus: Server [brian-test.unl.edu]
declared: Invalid request; user not authenticated(error code: 3006)
_______________________________________________________
Reply to this item at:
<http://savannah.cern.ch/bugs/?91129>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
