Hi Andy,
See below (seg fault happens in thread #10). Appears to be a race condition - one thread is starting up and using the static instance of type XrdNet before XrdConfig::Configure has finished.
If I restart the server enough times, it seems to be stable if it makes it past this hiccup.
I will be watching the git commits and rebuild/redeploy the RPM (with all the other fixes Lukasz did for me) once this fix is in.
Brian
On Sep 16, 2011, at 8:30 PM, Andrew Hanushevsky wrote:
> Any chance of getting this traceback with debug symbols?
>
> On Fri, 16 Sep 2011, Brian Bockelman wrote:
>
>> Ah - well, it segfaults on startup for the redirector. Traceback below. Will hit this up Monday.
>>
>> Brian
>>
>> (gdb) bt
>> #0 0x00002aaaab1a66a0 in XrdNet::Trim(char*) () from /usr/lib64/libXrdUtils.so.0
>> #1 0x00002aaaab1c47eb in XrdLink::Alloc(XrdNetPeer&, int) () from /usr/lib64/libXrdUtils.so.0
>> #2 0x00002aaaab1c0c61 in XrdInet::Connect(char const*, int, int, int) () from /usr/lib64/libXrdUtils.so.0
>> #3 0x00002aaaaab075de in XrdCmsClientMan::Hookup() () from /usr/lib64/libXrdServer.so.0
>> #4 0x00002aaaaab07aa9 in XrdCmsClientMan::Start() () from /usr/lib64/libXrdServer.so.0
>> #5 0x00002aaaaab0a7ef in XrdCmsStartManager(void*) () from /usr/lib64/libXrdServer.so.0
>> #6 0x00002aaaab18a93b in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.0
>> #7 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
>> #8 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
>>
>> On Sep 16, 2011, at 6:45 PM, Lukasz Janyst wrote:
>>
>>> Hi Brian,
>>>
>>> I was planning to cut -rc1 in around two weeks. It looks like
>>> XRootD will be pulled out of ROOT in the next release and we have to
>>> iron out the integration part.
>>>
>>> Lukasz
>>>
>>> 2011/9/17 Brian Bockelman <[log in to unmask]>:
>>>> Hi folks,
>>>>
>>>> Is it time yet to cut a pre-release for 3.1? I'm placing a RPM generated from git HEAD into our testing systems to see how it holds up over the weekend. It looks like it's far enough along to at least get it in the hands of plugin developers.
>>>>
>>>> Thanks for Lukasz for addressing my nitpicks from the first round of testing!
>>>>
>>>> Brian
>>>>
>>
>>
(gdb) thread apply all bt
[New Thread 0x40a09940 (LWP 12265)]
Thread 11 (Thread 0x40a09940 (LWP 12265)):
#0 0x0000003d29ad3f31 in clone () from /lib64/libc.so.6
#1 0x0000003d2a206660 in ?? () from /lib64/libpthread.so.0
#2 0x0000000040a09940 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x40908940 (LWP 12264)):
#0 0x00002aaaab1a76a0 in XrdNet::Trim (this=0x0, hname=0x64f09c "xrootd-itb") at /builddir/build/BUILD/xrootd/xrootd/src/XrdNet/XrdNet.cc:258
#1 0x00002aaaab1c57eb in XrdLink::Alloc (Peer=..., opts=0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdLink.cc:232
#2 0x00002aaaab1c1c61 in XrdInet::Connect (this=0x64a220, host=0x64b1c0 "xrootd-itb.unl.edu", port=1213, opts=0, tmo=-1) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdInet.cc:84
#3 0x00002aaaaab085de in XrdCmsClientMan::Hookup (this=0x64d090) at /builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsClientMan.cc:300
#4 0x00002aaaaab08aa9 in XrdCmsClientMan::Start (this=0x64d090) at /builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsClientMan.cc:202
#5 0x00002aaaaab0b7ef in XrdCmsStartManager (carg=0x64d090) at /builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:597
#6 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x64b2b0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#7 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x40807940 (LWP 12263)):
#0 0x0000003d2a20e1c1 in nanosleep () from /lib64/libpthread.so.0
#1 0x00002aaaaab00d59 in XrdAccConfig_Refresh (start_data=0x2aaaaad2b130) at /builddir/build/BUILD/xrootd/xrootd/src/XrdAcc/XrdAccConfig.cc:85
#2 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x64ae50) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#3 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x40706940 (LWP 12262)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x649330, syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at /builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x647b50) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x40605940 (LWP 12261)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x648730, syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at /builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6479f0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x40504940 (LWP 12260)):
#0 0x0000003d29acb186 in poll () from /lib64/libc.so.6
#1 0x00002aaaab1c80b5 in XrdPollPoll::Start (this=0x6441d0, syncsem=0x7fffffffb5c0, retcode=@0x7fffffffb5b8) at /builddir/build/BUILD/xrootd/xrootd/src/./Xrd/XrdPollPoll.icc:261
#2 0x00002aaaab1c66be in XrdStartPolling (parg=0x7fffffffb5b0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdPoll.cc:87
#3 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x647890) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#4 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x40403940 (LWP 12259)):
#0 0x0000003d2a20cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x000000000042dd97 in XrdSysSemaphore::Wait (this=0x2aaaaaf3c3e0) at /builddir/build/BUILD/xrootd/xrootd/src/./XrdSys/XrdSysPthread.hh:317
#2 0x00002aaaab1cb0f6 in XrdScheduler::Run (this=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:260
#3 0x00002aaaab1cb3bf in XrdStartWorking (carg=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:65
---Type <return> to continue, or q <return> to quit---
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x645310) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x40302940 (LWP 12258)):
#0 0x0000003d2a20cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x000000000042dd97 in XrdSysSemaphore::Wait (this=0x2aaaaaf3c3e0) at /builddir/build/BUILD/xrootd/xrootd/src/./XrdSys/XrdSysPthread.hh:317
#2 0x00002aaaab1cb0f6 in XrdScheduler::Run (this=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:260
#3 0x00002aaaab1cb3bf in XrdStartWorking (carg=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:65
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6451b0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x40201940 (LWP 12257)):
#0 0x0000003d2a20b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaab18b83a in XrdSysCondVar::Wait (this=0x2aaaaaf3c430, sec=780) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:130
#2 0x00002aaaab1caecf in XrdScheduler::TimeSched (this=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:525
#3 0x00002aaaab1caf33 in XrdStartTSched (carg=0x2aaaaaf3c340) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdScheduler.cc:59
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x645050) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x40100940 (LWP 12256)):
#0 0x0000003d2a20b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaab18b83a in XrdSysCondVar::Wait (this=0x2aaaaaf3c5f8, sec=1200) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:130
#2 0x00002aaaab1c0fd0 in XrdBuffManager::Reshape (this=0x2aaaaaf3c4f0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdBuffer.cc:205
#3 0x00002aaaab1c144b in XrdReshaper (pp=0x2aaaaaf3c4f0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdBuffer.cc:35
#4 0x00002aaaab18b93b in XrdSysThread_Xeq (myargs=0x6441a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:87
#5 0x0000003d2a20673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003d29ad3f6d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2aaaab3e44a0 (LWP 12253)):
#0 0x0000003d29ad3f31 in clone () from /lib64/libc.so.6
#1 0x0000003d2a20719f in pthread_create@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#2 0x00002aaaab18b616 in XrdSysThread::Run (tid=0x7fffffff9a18, proc=0x2aaaaab0b7ba <XrdCmsStartResp(void*)>, arg=0x0, opts=0, tDesc=0x2aaaaab1738a "async callback")
at /builddir/build/BUILD/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:283
#3 0x00002aaaaab0b76c in XrdCmsFinderRMT::StartManagers (this=0x64b030, myManList=0x64b1e0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:652
#4 0x00002aaaaab0d3ab in XrdCmsFinderRMT::Configure (this=0x64b030, cfn=0x64bb00 "/etc/xrootd/xrootd.cf", envP=0x7fffffffb2a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsFinder.cc:153
#5 0x00002aaaaaf58111 in XrdOfs::ConfigRedir (this=0x2aaaab168e60, Eroute=..., EnvInfo=0x7fffffffb2a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsConfig.cc:452
#6 0x00002aaaaaf59bc4 in XrdOfs::Configure (this=0x2aaaab168e60, Eroute=..., EnvInfo=0x7fffffffb2a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsConfig.cc:197
#7 0x00002aaaaaf55ab0 in XrdSfsGetDefaultFileSystem (native_fs=0x0, lp=0x2aaaaaf3c298, configfn=0x6391e0 "/etc/xrootd/xrootd.cf", EnvInfo=0x7fffffffb2a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdOfs/XrdOfsFS.cc:45
#8 0x00000000004161c1 in XrdXrootdProtocol::Configure (parms=0x0, pi=0x2aaaaaf3c1a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdXrootd/XrdXrootdConfig.cc:234
#9 0x0000000000420aae in XrdgetProtocol (pname=0x643d60 "xrootd", parms=0x0, pi=0x2aaaaaf3c1a0) at /builddir/build/BUILD/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:131
#10 0x00002aaaab1c959f in XrdProtLoad::getProtocol (lname=0x0, pname=0x643d60 "xrootd", parms=0x0, pi=0x2aaaaaf3c1a0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdProtLoad.cc:231
#11 0x00002aaaab1c9998 in XrdProtLoad::Load (lname=0x0, pname=0x643d60 "xrootd", parms=0x0, pi=0x2aaaaaf3c1a0) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdProtLoad.cc:78
#12 0x00002aaaaad36d3b in XrdConfig::Setup (this=0x2aaaaaf3c1a0, dfltp=0x7fffffffdc66 "xrootd") at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdConfig.cc:743
#13 0x00002aaaaad37dba in XrdConfig::Configure (this=0x2aaaaaf3c1a0, argc=8, argv=0x7fffffffbc68) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdConfig.cc:385
#14 0x00002aaaaad39551 in main (argc=8, argv=0x7fffffffbc68) at /builddir/build/BUILD/xrootd/xrootd/src/Xrd/XrdMain.cc:154
#15 0x0000003d29a1d994 in __libc_start_main () from /lib64/libc.so.6
#16 0x000000000040e4f9 in _start ()
(gdb) q
|