Just pushed a fix for this.
-----Original Message-----
From: Brian Bockelman
Sent: Monday, September 19, 2011 9:25 AM
To: Andrew Hanushevsky
Cc: Lukasz Janyst ; xrootd-dev
Subject: Re: 3.1 pre-release?
Hi Andy,
See below (seg fault happens in thread #10). Appears to be a race
condition - one thread is starting up and using the static instance of type
XrdNet before XrdConfig::Configure has finished.
If I restart the server enough times, it seems to be stable if it makes it
past this hiccup.
I will be watching the git commits and rebuild/redeploy the RPM (with all
the other fixes Lukasz did for me) once this fix is in.
Brian
On Sep 16, 2011, at 8:30 PM, Andrew Hanushevsky wrote:
> Any chance of getting this traceback with debug symbols?
>
> On Fri, 16 Sep 2011, Brian Bockelman wrote:
>
>> Ah - well, it segfaults on startup for the redirector. Traceback below.
>> Will hit this up Monday.
>>
>> Brian
>>
>> (gdb) bt
>> #0 0x00002aaaab1a66a0 in XrdNet::Trim(char*) () from
>> /usr/lib64/libXrdUtils.so.0
>> #1 0x00002aaaab1c47eb in XrdLink::Alloc(XrdNetPeer&, int) () from
>> /usr/lib64/libXrdUtils.so.0
>> #2 0x00002aaaab1c0c61 in XrdInet::Connect(char const*, int, int, int) ()
>> from /usr/lib64/libXrdUtils.so.0
>> #3 0x00002aaaaab075de in XrdCmsClientMan::Hookup() () from
>> /usr/lib64/libXrdServer.so.0
>> #4 0x00002aaaaab07aa9 in XrdCmsClientMan::Start() () from
>> /usr/lib64/libXrdServer.so.0
>> #5 0x00002aaaaab0a7ef in XrdCmsStartManager(void*) () from
>> /usr/lib64/libXrdServer.so.0
>> #6 0x00002aaaab18a93b in XrdSysThread_Xeq () from
>> /usr/lib64/libXrdUtils.so.0
>> #7 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
>> #8 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
>>
>> On Sep 16, 2011, at 6:45 PM, Lukasz Janyst wrote:
>>
>>> Hi Brian,
>>>
>>> I was planning to cut -rc1 in around two weeks. It looks like
>>> XRootD will be pulled out of ROOT in the next release and we have to
>>> iron out the integration part.
>>>
>>> Lukasz
>>>
>>> 2011/9/17 Brian Bockelman <[log in to unmask]>:
>>>> Hi folks,
>>>>
>>>> Is it time yet to cut a pre-release for 3.1? I'm placing a RPM
>>>> generated from git HEAD into our testing systems to see how it holds up
>>>> over the weekend. It looks like it's far enough along to at least get
>>>> it in the hands of plugin developers.
>>>>
>>>> Thanks for Lukasz for addressing my nitpicks from the first round of
>>>> testing!
>>>>
>>>> Brian
>>>>
>>
>>
(gdb) thread apply all bt
[New Thread 0x40a09940 (LWP 12265)]
Thread 11 (Thread 0x40a09940 (LWP 12265)):
#0 0x0000003d29ad3f31 in clone () from /lib64/libc.so.6
#1 0x0000003d2a206660 in ?? () from /lib64/libpthread.so.0
#2 0x0000000040a09940 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x40908940 (LWP 12264)):
#0 0x00002aaaab1a76a0 in XrdNet::Trim (this=0x0, hname=0x64f09c
"xrootd-itb") at
/builddir/build/BUILD/xrootd/xrootd/src/XrdNet/XrdNet.cc:258
#1 0x00002aaaab1c57eb in XrdLink::Alloc (Peer=..., opts=0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdLink.cc:232
#2 0x00002aaaab1c1c61 in XrdInet::Connect (this=0x64a220, host=0x64b1c0
"xrootd-itb.unl.edu", port=1213, opts=0, tmo=-1) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdInet.cc:84
#3 0x00002aaaaab085de in XrdCmsClientMan::Hookup (this=0x64d090) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsClientMan.cc:300
#4 0x00002aaaaab08aa9 in XrdCmsClientMan::Start (this=0x64d090) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsClientMan.cc:202
#5 0x00002aaaaab0b7ef in XrdCmsStartManager (carg=0x64d090) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:597
#6 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x64b2b0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#7 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x40807940 (LWP 12263)):
#0 0x0000003d2a20e1c1 in nanosleep () from /lib64/libpthread.so.0
#1 0x00002aaaaab00d59 in XrdAccConfig_Refresh (start_data=0x2aaaaad2b130)
at /builddir/build/BUILD/xrootd/xrootd/src/XrdAcc/XrdAccConfig.cc:85
#2 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x64ae50) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#3 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x40706940 (LWP 12262)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x649330,
syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at
/builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x647b50) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x40605940 (LWP 12261)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x648730,
syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at
/builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6479f0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x40504940 (LWP 12260)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x6441d0,
syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at
/builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x647890) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x40403940 (LWP 12259)):
#0 0x0000003d2a20cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x000000000042dd97 in XrdSysSemaphore::Wait (this=0x2aaaaaf3c3e0) at
/builddir/build/BUILD/xrootd/xrootd/src/./XrdSys/XrdSysPthread.hh:317
#2 0x00002aaaab1cb0f6 in XrdScheduler::Run (this=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:260
#3 0x00002aaaab1cb3bf in XrdStartWorking (carg=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:65
---Type <return> to continue, or q <return> to quit---
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x645310) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x40302940 (LWP 12258)):
#0 0x0000003d2a20cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x000000000042dd97 in XrdSysSemaphore::Wait (this=0x2aaaaaf3c3e0) at
/builddir/build/BUILD/xrootd/xrootd/src/./XrdSys/XrdSysPthread.hh:317
#2 0x00002aaaab1cb0f6 in XrdScheduler::Run (this=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:260
#3 0x00002aaaab1cb3bf in XrdStartWorking (carg=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:65
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6451b0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x40201940 (LWP 12257)):
#0 0x0000003d2a20b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002aaaab18b83a in XrdSysCondVar::Wait (this=0x2aaaaaf3c430, sec=780)
at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:130
#2 0x00002aaaab1caecf in XrdScheduler::TimeSched (this=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:525
#3 0x00002aaaab1caf33 in XrdStartTSched (carg=0x2aaaaaf3c340) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:59
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x645050) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x40100940 (LWP 12256)):
#0 0x0000003d2a20b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002aaaab18b83a in XrdSysCondVar::Wait (this=0x2aaaaaf3c5f8,
sec=1200) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:130
#2 0x00002aaaab1c0fd0 in XrdBuffManager::Reshape (this=0x2aaaaaf3c4f0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdBuffer.cc:205
#3 0x00002aaaab1c144b in XrdReshaper (pp=0x2aaaaaf3c4f0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdBuffer.cc:35
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6441a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2aaaab3e44a0 (LWP 12253)):
#0 0x0000003d29ad3f31 in clone () from /lib64/libc.so.6
#1 0x0000003d2a20719f in pthread_create@@GLIBC_2.2.5 () from
/lib64/libpthread.so.0
#2 0x00002aaaab18b616 in XrdSysThread::Run (tid=0x7fffffff9a18,
proc=0x2aaaaab0b7ba <XrdCmsStartResp(void*)>, arg=0x0, opts=0,
tDesc=0x2aaaaab1738a "async callback")
at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:283
#3 0x00002aaaaab0b76c in XrdCmsFinderRMT::StartManagers (this=0x64b030,
myManList=0x64b1e0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:652
#4 0x00002aaaaab0d3ab in XrdCmsFinderRMT::Configure (this=0x64b030,
cfn=0x64bb00 "/etc/xrootd/xrootd.cf", envP=0x7fffffffb2a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:153
#5 0x00002aaaaaf58111 in XrdOfs::ConfigRedir (this=0x2aaaab168e60,
Eroute=..., EnvInfo=0x7fffffffb2a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsConfig.cc:452
#6 0x00002aaaaaf59bc4 in XrdOfs::Configure (this=0x2aaaab168e60,
Eroute=..., EnvInfo=0x7fffffffb2a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsConfig.cc:197
#7 0x00002aaaaaf55ab0 in XrdSfsGetDefaultFileSystem (native_fs=0x0,
lp=0x2aaaaaf3c298, configfn=0x6391e0 "/etc/xrootd/xrootd.cf",
EnvInfo=0x7fffffffb2a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsFS.cc:45
#8 0x00000000004161c1 in XrdXrootdProtocol::Configure (parms=0x0,
pi=0x2aaaaaf3c1a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdXrootd/XrdXrootdConfig.cc:234
#9 0x0000000000420aae in XrdgetProtocol (pname=0x643d60 "xrootd",
parms=0x0, pi=0x2aaaaaf3c1a0) at
/builddir/build/BUILD/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:131
#10 0x00002aaaab1c959f in XrdProtLoad::getProtocol (lname=0x0,
pname=0x643d60 "xrootd", parms=0x0, pi=0x2aaaaaf3c1a0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdProtLoad.cc:231
#11 0x00002aaaab1c9998 in XrdProtLoad::Load (lname=0x0, pname=0x643d60
"xrootd", parms=0x0, pi=0x2aaaaaf3c1a0) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdProtLoad.cc:78
#12 0x00002aaaaad36d3b in XrdConfig::Setup (this=0x2aaaaaf3c1a0,
dfltp=0x7fffffffdc66 "xrootd") at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdConfig.cc:743
#13 0x00002aaaaad37dba in XrdConfig::Configure (this=0x2aaaaaf3c1a0, argc=8,
argv=0x7fffffffbc68) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdConfig.cc:385
#14 0x00002aaaaad39551 in main (argc=8, argv=0x7fffffffbc68) at
/builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdMain.cc:154
#15 0x0000003d29a1d994 in __libc_start_main () from /lib64/libc.so.6
#16 0x000000000040e4f9 in _start ()
(gdb) q
