Hi,
I successfully got a core from one of the working servers. Here's the issue:
1) Client out in Europe (probably behind a misconfigured firewall) requests a ton of data and abruptly dies without closing socket; socket queue fills; writing blocks because the send queue is full:
[root@red-gridftp6 ~]# netstat -tnap | grep 57530
tcp 6720 730552 129.93.239.130:1094 193.146.75.180:57530 ESTABLISHED 5474/xrootd
730552 bytes are in the send queue.
[root@red-gridftp6 ~]# strace -p 5524
Process 5524 attached - interrupt to quit
write(79, "\304\367\34\255x\240IF\3\224).\3472\255=TL\335%m\240\225\362\22a\343TP\356\276\230"..., 220646 <unfinished ...>
Process 5524 detached
(waited about 30 seconds before hit Ctrl+C)
2) Thread 5524 is holding the write mutex:
(gdb) bt
#0 0x0000003de940d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003de9408e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003de9408cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0000000000439230 in Lock (this=0x15da9198, ctime=0x0) at ../../src/XrdSys/XrdSysPthread.hh:132
#4 XrdLink::syncStats (this=0x15da9198, ctime=0x0) at XrdLink.cc:1114
#5 0x000000000043a5c1 in XrdLink::Stats (buff=0x2aaab011713f "", blen=2474, do_sync=<value optimized out>) at XrdLink.cc:1081
#6 0x00000000004400f5 in XrdStats::Stats (this=0x1467a1d0, opts=-1) at XrdStats.cc:184
#7 0x00000000004403d4 in XrdStats::Report (this=0x1467a1d0, Dest=0x0, iVal=600, Opts=<value optimized out>) at XrdStats.cc:124
#8 0x0000000000440496 in XrdStatsJob::DoIt (this=0x14fb3c00) at XrdStats.cc:52
#9 0x000000000043e9a8 in XrdScheduler::Run (this=0x6a4dc0) at XrdScheduler.cc:292
#10 0x000000000043eb19 in XrdStartWorking (carg=0x15da9370) at XrdScheduler.cc:73
#11 0x0000000000481d17 in XrdSysThread_Xeq (myargs=<value optimized out>) at XrdSysPthread.cc:87
#12 0x0000003de940673d in start_thread () from /lib64/libpthread.so.0
#13 0x0000003de88d3f6d in clone () from /lib64/libc.so.6
(gdb) up 4
#4 XrdLink::syncStats (this=0x15da9198, ctime=0x0) at XrdLink.cc:1114
1114 wrMutex.Lock();
(gdb) p this->wrMutex
$9 = {cs = {__data = {__lock = 2, __count = 0, __owner = 5524, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\224\025\000\000\001", '\000' <repeats 26 times>,
__align = 2}}
3) Thread 32118 is in the middle of updating the stats; holds the global statsMutex and is blocking on the wrMutex held by thread 5524
(gdb) bt
#0 0x0000003de88ccd83 in writev () from /lib64/libc.so.6
#1 0x000000000043aa06 in XrdLink::Send (this=0x15a733e8, iov=0x15b6a290, iocnt=2, bytes=14726) at XrdLink.cc:730
#2 0x0000000000430004 in XrdXrootdResponse::Send (this=0x15b6a280, data=<value optimized out>, dlen=35249) at XrdXrootdResponse.cc:146
#3 0x0000000000434925 in XrdXrootdProtocol::do_Read (this=0x15b6a060) at XrdXrootdXeq.cc:1530
#4 0x0000000000439f80 in XrdLink::DoIt (this=0x15a733e8) at XrdLink.cc:419
#5 0x000000000043e9a8 in XrdScheduler::Run (this=0x6a4dc0) at XrdScheduler.cc:292
#6 0x000000000043eb19 in XrdStartWorking (carg=0x52) at XrdScheduler.cc:73
#7 0x0000000000481d17 in XrdSysThread_Xeq (myargs=<value optimized out>) at XrdSysPthread.cc:87
#8 0x0000003de940673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003de88d3f6d in clone () from /lib64/libc.so.6
(gdb) up 1
#1 0x000000000043aa06 in XrdLink::Send (this=0x15a733e8, iov=0x15b6a290, iocnt=2, bytes=14726) at XrdLink.cc:730
730 {do {retc = writev(FD, iov, iocnt);} while(retc < 0 && errno == EINTR);
(gdb) p this->wrMutex
$7 = {cs = {__data = {__lock = 1, __count = 0, __owner = 32118, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000v}\000\000\001", '\000' <repeats 26 times>,
__align = 1}}
4) Main thread is trying to grab the statsMutex, but it is held by 32118.
Thread 42 (Thread 5474):
#0 0x0000003de940d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003de9408e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003de9408cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043bb26 in Lock (Peer=..., opts=0) at ../../src/XrdSys/XrdSysPthread.hh:132
#4 XrdLink::Alloc (Peer=..., opts=0) at XrdLink.cc:247
#5 0x0000000000443a28 in XrdInet::Accept (this=0x1467bad0, opts=<value optimized out>, timeout=<value optimized out>) at XrdInet.cc:58
#6 0x000000000043c4d2 in mainAccept (parg=<value optimized out>) at XrdMain.cc:101
#7 0x000000000043c673 in main (argc=9, argv=0x7fff3cd5ffa8) at XrdMain.cc:186
So, because one client in Spain has a nasty firewall, the xrootd server in Nebraska stops accepting connections.
Ideas?
Brian
On Mar 3, 2011, at 12:46 PM, Brian Bockelman wrote:
> Hi,
>
> In the same stress test I'm running, I've got about 5 servers that stopped accepting clients. Stack trace below. I accidentally screwed up the process, so no core is available.
>
> However, I was able to get line numbers:
>
> (gdb) bt
> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x000000000043bb26 in Lock (Peer=..., opts=0) at ../../src/XrdSys/XrdSysPthread.hh:132
> #4 XrdLink::Alloc (Peer=..., opts=0) at XrdLink.cc:247
> #5 0x0000000000443a28 in XrdInet::Accept (this=0x14204b10, opts=<value optimized out>, timeout=<value optimized out>) at XrdInet.cc:58
> #6 0x000000000043c4d2 in mainAccept (parg=<value optimized out>) at XrdMain.cc:101
> #7 0x000000000043c673 in main (argc=9, argv=0x7ffff40c04f8) at XrdMain.cc:186
>
> The relevant lines from XrdLink.cc are:
>
>
> // Return the link
> //
> statsMutex.Lock(); <-- line 247
> LinkCountTot++;
> if (LinkCountMax == LinkCount++) LinkCountMax = LinkCount;
> statsMutex.UnLock();
>
> So - somewhere, the statsMutex got locked and never unlocked.
>
> Ideas?
>
> Brian
>
> Thread 43 (Thread 0x4091b940 (LWP 4286)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
> #2 0x0000000000438881 in XrdBuffManager::Reshape() ()
> #3 0x0000000000438b99 in XrdReshaper(void*) ()
> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 42 (Thread 0x41923940 (LWP 4287)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
> #2 0x000000000043fab2 in XrdScheduler::TimeSched() ()
> #3 0x000000000043fb19 in XrdStartTSched(void*) ()
> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 41 (Thread 0x41a24940 (LWP 4288)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 40 (Thread 0x41b25940 (LWP 4289)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 39 (Thread 0x40168940 (LWP 4290)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 38 (Thread 0x41785940 (LWP 4291)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 37 (Thread 0x40ed8940 (LWP 4293)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 36 (Thread 0x405cc940 (LWP 4295)):
> #0 0x000000355020e1c1 in nanosleep () from /lib64/libpthread.so.0
> #1 0x000000000045592a in XrdAccConfig_Refresh(void*) ()
> #2 0x0000000000481d17 in XrdSysThread_Xeq ()
> #3 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #4 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 35 (Thread 0x40a1c940 (LWP 4296)):
> #0 0x000000355020d91b in read () from /lib64/libpthread.so.0
> #1 0x000000000047cd60 in XrdOucStream::GetLine() ()
> #2 0x00000000004479a8 in XrdOfsEvr::recvEvents() ()
> #3 0x0000000000447a69 in XrdOfsEvRecv(void*) ()
> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 34 (Thread 0x41611940 (LWP 4297)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x0000000000447b2f in XrdOfsEvr::flushEvents() ()
> #2 0x0000000000447bd9 in XrdOfsEvFlush(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 33 (Thread 0x40fff940 (LWP 4298)):
> #0 0x000000355020dc81 in recv () from /lib64/libpthread.so.0
> #1 0x000000000046da37 in XrdCmsFinderTRG::Start() ()
> #2 0x0000000000481d17 in XrdSysThread_Xeq ()
> #3 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #4 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 32 (Thread 0x41e8c940 (LWP 4300)):
> #0 0x000000355020db3b in accept () from /lib64/libpthread.so.0
> #1 0x0000000000477e24 in XrdNetSocket::Accept(int) ()
> #2 0x0000000000422e0d in XrdXrootdAdmin::Start(XrdNetSocket*) ()
> #3 0x0000000000423819 in XrdXrootdInitAdmin(void*) ()
> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 31 (Thread 0x41f8d940 (LWP 4317)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 30 (Thread 0x4208e940 (LWP 4318)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 29 (Thread 0x4218f940 (LWP 4319)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 28 (Thread 0x42290940 (LWP 4320)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 27 (Thread 0x41d3a940 (LWP 4321)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7956 in os::PlatformEvent::park(long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7f09b in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaad0ee493 in VMThread::loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaad0ee08e in VMThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 26 (Thread 0x42391940 (LWP 4322)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaad06bf2a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaad069293 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaace01b7e in JVM_MonitorWait () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaab4010c88 in ?? ()
> #6 0x00000000151e5800 in ?? ()
> #7 0x00002aaaad068eb5 in ObjectSynchronizer::fast_exit(oopDesc*, BasicLock*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #8 0x00002aaab41a81a8 in ?? ()
> #9 0x0000000000000000 in ?? ()
> Thread 25 (Thread 0x42492940 (LWP 4323)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaad06bf2a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaad069293 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaace01b7e in JVM_MonitorWait () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaab4010c88 in ?? ()
> #6 0x00000000151ed000 in ?? ()
> #7 0x00000000151ed878 in ?? ()
> #8 0x00000000151ed000 in ?? ()
> #9 0x0000000042491a08 in ?? ()
> #10 0x0000000000000000 in ?? ()
> Thread 24 (Thread 0x4045c940 (LWP 4324)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa84a5 in check_pending_signals(bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacfa11c7 in signal_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 23 (Thread 0x40c50940 (LWP 4325)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f5cd in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacc68bf3 in CompileQueue::get() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacc6a781 in CompileBroker::compiler_thread_loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaad0a4969 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #8 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #9 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 22 (Thread 0x41c26940 (LWP 4326)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f5cd in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacc68bf3 in CompileQueue::get() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaacc6a781 in CompileBroker::compiler_thread_loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaad0a4969 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #8 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #9 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 21 (Thread 0x42593940 (LWP 4327)):
> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacf28ec2 in LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 20 (Thread 0x42694940 (LWP 4328)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x00002aaaacfa7956 in os::PlatformEvent::park(long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #2 0x00002aaaacfa5b5b in os::sleep(Thread*, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #3 0x00002aaaad09d54b in WatcherThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #4 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 19 (Thread 0x42795940 (LWP 4329)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #3 0x00002aaab4157072 in ?? ()
> #4 0x00002aaabfdc7750 in ?? ()
> #5 0x00002aaab41358e6 in ?? ()
> #6 0x00002aab6519ab40 in ?? ()
> #7 0x00002aaab7953550 in ?? ()
> #8 0x00002aaabfdc7420 in ?? ()
> #9 0x00002aaab4136618 in ?? ()
> #10 0x0000000042794470 in ?? ()
> #11 0x0000000014cd0800 in ?? ()
> #12 0x0000000000000001 in ?? ()
> #13 0x00002aaab4150634 in ?? ()
> #14 0x00002aaabfdc7750 in ?? ()
> #15 0x00002aaa00000001 in ?? ()
> #16 0x000000000000ea60 in ?? ()
> #17 0x00002aaabfdc7888 in ?? ()
> #18 0x0000000000000007 in ?? ()
> #19 0x00002aaabfdc7750 in ?? ()
> #20 0x0000000000000009 in ?? ()
> #21 0x00000000427943a8 in ?? ()
> #22 0x00002aaabc4c0ea8 in ?? ()
> #23 0x00002aaab4161740 in ?? ()
> #24 0x00002aab6519ab40 in ?? ()
> #25 0x00002aaabfdc7390 in ?? ()
> #26 0x00002aaabfdc7720 in ?? ()
> #27 0x000000000000ea60 in ?? ()
> #28 0x00002aaabfdc7738 in ?? ()
> #29 0x00002aaabfdc7390 in ?? ()
> #30 0x00002aaabfdc7720 in ?? ()
> #31 0x00002aaabfdc7390 in ?? ()
> #32 0x00002aaabfdc7738 in ?? ()
> #33 0x00002aab651970f8 in ?? ()
> #34 0x00002aab6519ab40 in ?? ()
> #35 0x00002aab65197028 in ?? ()
> #36 0x0000000000000009 in ?? ()
> #37 0x0000000000000009 in ?? ()
> #38 0x0000000000000009 in ?? ()
> #39 0x0000000000000001 in ?? ()
> #40 0x0000000000000009 in ?? ()
> #41 0x0000000000099473 in ?? ()
> #42 0x0000000000000000 in ?? ()
> Thread 18 (Thread 0x406cd940 (LWP 802)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 17 (Thread 0x42896940 (LWP 872)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 16 (Thread 0x42997940 (LWP 1695)):
> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 15 (Thread 0x41302940 (LWP 1789)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 14 (Thread 0x407ce940 (LWP 1935)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 13 (Thread 0x40b1d940 (LWP 2272)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #3 0x00002aaab4157072 in ?? ()
> #4 0x00002aaabfb2eba0 in ?? ()
> #5 0x00002aaab41358e6 in ?? ()
> #6 0x00002aab649340f0 in ?? ()
> #7 0x00002aaab4155e0c in ?? ()
> #8 0x00002aaabfb2e870 in ?? ()
> #9 0x00002aaab4136618 in ?? ()
> #10 0x00002aaac0087b08 in ?? ()
> #11 0x00002aaab4136618 in ?? ()
> #12 0x0000000000000001 in ?? ()
> #13 0x00002aaab4150634 in ?? ()
> #14 0x00002aaabfb2eba0 in ?? ()
> #15 0x00002aaa00000001 in ?? ()
> #16 0x000000000000ea60 in ?? ()
> #17 0x00002aaabfb2ecd8 in ?? ()
> #18 0x0000000000000007 in ?? ()
> #19 0x00002aaabfb2eba0 in ?? ()
> #20 0x0000000000000009 in ?? ()
> #21 0x00002aaab70c8050 in ?? ()
> #22 0x00002aaabc4c0ea8 in ?? ()
> #23 0x00002aaab4161740 in ?? ()
> #24 0x00002aab649340f0 in ?? ()
> #25 0x00002aaabfb2e7e0 in ?? ()
> #26 0x00002aaabfb2eb70 in ?? ()
> #27 0x000000000000ea60 in ?? ()
> #28 0x00002aaabfb2eb88 in ?? ()
> #29 0x00002aaabfb2e7e0 in ?? ()
> #30 0x00002aaabfb2eb70 in ?? ()
> #31 0x00002aaabfb2e7e0 in ?? ()
> #32 0x00002aaabfb2eb88 in ?? ()
> #33 0x00002aab64928620 in ?? ()
> #34 0x00002aab649340f0 in ?? ()
> #35 0x00002aab64928550 in ?? ()
> #36 0x0000000000000009 in ?? ()
> #37 0x0000000000000009 in ?? ()
> #38 0x0000000000000009 in ?? ()
> #39 0x0000000000000001 in ?? ()
> #40 0x0000000000000009 in ?? ()
> #41 0x0000000000045367 in ?? ()
> #42 0x0000000000000000 in ?? ()
> Thread 12 (Thread 0x41100940 (LWP 3448)):
> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
> #3 0x00002aaab4157072 in ?? ()
> #4 0x00002aaabf9e8448 in ?? ()
> #5 0x00002aaab41358e6 in ?? ()
> #6 0x00002aab6474b9b8 in ?? ()
> #7 0x00002aaab4155e0c in ?? ()
> #8 0x00002aaabf9e8118 in ?? ()
> #9 0x00002aaab4136618 in ?? ()
> #10 0x00002aaabfb2e870 in ?? ()
> #11 0x00002aaab4136618 in ?? ()
> #12 0x0000000000000001 in ?? ()
> #13 0x00002aaab4150634 in ?? ()
> #14 0x00002aaabf9e8448 in ?? ()
> #15 0x00002aaa00000001 in ?? ()
> #16 0x000000000000ea60 in ?? ()
> #17 0x00002aaabf9e8580 in ?? ()
> #18 0x0000000000000007 in ?? ()
> #19 0x00002aaabf9e8448 in ?? ()
> #20 0x0000000000000009 in ?? ()
> #21 0x00002aaab70c8050 in ?? ()
> #22 0x00002aaabc4c0ea8 in ?? ()
> #23 0x00002aaab4161740 in ?? ()
> #24 0x00002aab6474b9b8 in ?? ()
> #25 0x00002aaabf9e8088 in ?? ()
> #26 0x00002aaabf9e8418 in ?? ()
> #27 0x000000000000ea60 in ?? ()
> #28 0x00002aaabf9e8430 in ?? ()
> #29 0x00002aaabf9e8088 in ?? ()
> #30 0x00002aaabf9e8418 in ?? ()
> #31 0x00002aaabf9e8088 in ?? ()
> #32 0x00002aaabf9e8430 in ?? ()
> #33 0x00002aab646041d8 in ?? ()
> #34 0x00002aab6474b9b8 in ?? ()
> #35 0x00002aab64604108 in ?? ()
> #36 0x0000000000000009 in ?? ()
> #37 0x0000000000000009 in ?? ()
> #38 0x0000000000000009 in ?? ()
> #39 0x0000000000000001 in ?? ()
> #40 0x0000000000000009 in ?? ()
> #41 0x0000000000043570 in ?? ()
> #42 0x0000000000000000 in ?? ()
> Thread 11 (Thread 0x41201940 (LWP 3552)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x000000000043e92e in XrdScheduler::Run() ()
> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 10 (Thread 0x41504940 (LWP 3862)):
> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 9 (Thread 0x42a98940 (LWP 3863)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x000000000043e92e in XrdScheduler::Run() ()
> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 8 (Thread 0x42b99940 (LWP 3864)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x000000000043e92e in XrdScheduler::Run() ()
> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 7 (Thread 0x42c9a940 (LWP 3865)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x000000000043e92e in XrdScheduler::Run() ()
> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 6 (Thread 0x42d9b940 (LWP 3866)):
> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 5 (Thread 0x42e9c940 (LWP 3867)):
> #0 0x000000355020d89b in write () from /lib64/libpthread.so.0
> #1 0x000000000043aac6 in XrdLink::Send(iovec const*, int, int) ()
> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 4 (Thread 0x42f9d940 (LWP 3868)):
> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x0000000000439230 in XrdLink::syncStats(int*) ()
> #4 0x000000000043a5c1 in XrdLink::Stats(char*, int, int) ()
> #5 0x00000000004400f5 in XrdStats::Stats(int) ()
> #6 0x00000000004403d4 in XrdStats::Report(char**, int, int) ()
> #7 0x0000000000440496 in XrdStatsJob::DoIt() ()
> #8 0x000000000043e9a8 in XrdScheduler::Run() ()
> #9 0x000000000043eb19 in XrdStartWorking(void*) ()
> #10 0x0000000000481d17 in XrdSysThread_Xeq ()
> #11 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x4309e940 (LWP 3869)):
> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
> #4 0x0000000000439f80 in XrdLink::DoIt() ()
> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x41403940 (LWP 5571)):
> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
> #1 0x000000000043e92e in XrdScheduler::Run() ()
> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x2b151fb5de20 (LWP 4285)):
> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x000000000043bb26 in XrdLink::Alloc(XrdNetPeer&, int) ()
> #4 0x0000000000443a28 in XrdInet::Accept(int, int) ()
> #5 0x000000000043c4d2 in mainAccept(void*) ()
> #6 0x000000000043c673 in main ()
> [root@red-gridftp4 ~]# pstack 4285 | head
> Thread 43 (Thread 0x4091b940 (LWP 4286)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
> #2 0x0000000000438881 in XrdBuffManager::Reshape() ()
> #3 0x0000000000438b99 in XrdReshaper(void*) ()
> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 42 (Thread 0x41923940 (LWP 4287)):
> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> [root@red-gridftp4 ~]# pstack 4285 | tail
> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x2b151fb5de20 (LWP 4285)):
> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x000000000043bb26 in XrdLink::Alloc(XrdNetPeer&, int) ()
> #4 0x0000000000443a28 in XrdInet::Accept(int, int) ()
> #5 0x000000000043c4d2 in mainAccept(void*) ()
> #6 0x000000000043c673 in main ()
>
