On Mar 3, 2011, at 2:53 PM, Andrew Hanushevsky wrote:
> Hi Brian,
>
> I agree with you that this is not good and should be fixed. We have two competing problems. You are running periodic statistics asking for synchronous accuracy. That requires the statsMutex to be held. It should be clear that to get valid I/O statistics, the thread also needs to obtain the writeMutex. Obviously, it's stuck holding one but waiting for the other. The main thread, incorrectly, assumes that no one will hold the statsMutex for any length of time. It needs the mutex to accurately count connections. So, yes, this is a problem.
>
One thought is to switch the order of the mutex acquisition: first acquire wrMutex (as it can block), then acquire statsMutex (very unlikely to block). This would at least allow you to continue to accept new clients, even if it swallows a thread.
> You're immediate bypass (until this is fixed) is to not ask for synchronous accuracy.
>
Howso?
> Would partial accuracy be OK for you? That is, the stats thread would try to get the lock a few times and if it fails it could either a) not include those statistics, or b) include them without getting the lock (less accurate). Which would you prefer?
>
Yes - at least, it should be able to timeout after a few milliseconds: can't it always get the statistics later?
Two additional thoughts:
1) I'm working on a git branch to switch all the statistics to atomic ops if they are builtin to the compiler (if not available, keeps the pthread). Willing to accept this patch?
2) I'm sort of surprised XrdLink uses blocking operations. Why not? Doing this, one could guarantee the maximum amount of time that wrMutex and rdMutex is held.
Brian
> Andy
>
> -----Original Message----- From: Brian Bockelman
> Sent: Thursday, March 03, 2011 11:24 AM
> To: Brian Bockelman
> Cc: xrootd-dev
> Subject: Re: Xrootd stops accepting clients due to stats mutex
>
> Addendum: I mixed up the threads in (2) and (3) backward in the diagnosis, which should be obvious if you read the stack traces.
>
> Brian
>
> On Mar 3, 2011, at 1:21 PM, Brian Bockelman wrote:
>
>> Hi,
>>
>> I successfully got a core from one of the working servers. Here's the issue:
>>
>> 1) Client out in Europe (probably behind a misconfigured firewall) requests a ton of data and abruptly dies without closing socket; socket queue fills; writing blocks because the send queue is full:
>>
>> [root@red-gridftp6 ~]# netstat -tnap | grep 57530
>> tcp 6720 730552 129.93.239.130:1094 193.146.75.180:57530 ESTABLISHED 5474/xrootd
>>
>> 730552 bytes are in the send queue.
>>
>> [root@red-gridftp6 ~]# strace -p 5524
>> Process 5524 attached - interrupt to quit
>> write(79, "\304\367\34\255x\240IF\3\224).\3472\255=TL\335%m\240\225\362\22a\343TP\356\276\230"..., 220646 <unfinished ...>
>> Process 5524 detached
>> (waited about 30 seconds before hit Ctrl+C)
>>
>> 2) Thread 5524 is holding the write mutex:
>>
>> (gdb) bt
>> #0 0x0000003de940d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>> #1 0x0000003de9408e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>> #2 0x0000003de9408cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>> #3 0x0000000000439230 in Lock (this=0x15da9198, ctime=0x0) at ../../src/XrdSys/XrdSysPthread.hh:132
>> #4 XrdLink::syncStats (this=0x15da9198, ctime=0x0) at XrdLink.cc:1114
>> #5 0x000000000043a5c1 in XrdLink::Stats (buff=0x2aaab011713f "", blen=2474, do_sync=<value optimized out>) at XrdLink.cc:1081
>> #6 0x00000000004400f5 in XrdStats::Stats (this=0x1467a1d0, opts=-1) at XrdStats.cc:184
>> #7 0x00000000004403d4 in XrdStats::Report (this=0x1467a1d0, Dest=0x0, iVal=600, Opts=<value optimized out>) at XrdStats.cc:124
>> #8 0x0000000000440496 in XrdStatsJob::DoIt (this=0x14fb3c00) at XrdStats.cc:52
>> #9 0x000000000043e9a8 in XrdScheduler::Run (this=0x6a4dc0) at XrdScheduler.cc:292
>> #10 0x000000000043eb19 in XrdStartWorking (carg=0x15da9370) at XrdScheduler.cc:73
>> #11 0x0000000000481d17 in XrdSysThread_Xeq (myargs=<value optimized out>) at XrdSysPthread.cc:87
>> #12 0x0000003de940673d in start_thread () from /lib64/libpthread.so.0
>> #13 0x0000003de88d3f6d in clone () from /lib64/libc.so.6
>> (gdb) up 4
>> #4 XrdLink::syncStats (this=0x15da9198, ctime=0x0) at XrdLink.cc:1114
>> 1114 wrMutex.Lock();
>> (gdb) p this->wrMutex
>> $9 = {cs = {__data = {__lock = 2, __count = 0, __owner = 5524, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\224\025\000\000\001", '\000' <repeats 26 times>,
>> __align = 2}}
>>
>> 3) Thread 32118 is in the middle of updating the stats; holds the global statsMutex and is blocking on the wrMutex held by thread 5524
>>
>> (gdb) bt
>> #0 0x0000003de88ccd83 in writev () from /lib64/libc.so.6
>> #1 0x000000000043aa06 in XrdLink::Send (this=0x15a733e8, iov=0x15b6a290, iocnt=2, bytes=14726) at XrdLink.cc:730
>> #2 0x0000000000430004 in XrdXrootdResponse::Send (this=0x15b6a280, data=<value optimized out>, dlen=35249) at XrdXrootdResponse.cc:146
>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read (this=0x15b6a060) at XrdXrootdXeq.cc:1530
>> #4 0x0000000000439f80 in XrdLink::DoIt (this=0x15a733e8) at XrdLink.cc:419
>> #5 0x000000000043e9a8 in XrdScheduler::Run (this=0x6a4dc0) at XrdScheduler.cc:292
>> #6 0x000000000043eb19 in XrdStartWorking (carg=0x52) at XrdScheduler.cc:73
>> #7 0x0000000000481d17 in XrdSysThread_Xeq (myargs=<value optimized out>) at XrdSysPthread.cc:87
>> #8 0x0000003de940673d in start_thread () from /lib64/libpthread.so.0
>> #9 0x0000003de88d3f6d in clone () from /lib64/libc.so.6
>> (gdb) up 1
>> #1 0x000000000043aa06 in XrdLink::Send (this=0x15a733e8, iov=0x15b6a290, iocnt=2, bytes=14726) at XrdLink.cc:730
>> 730 {do {retc = writev(FD, iov, iocnt);} while(retc < 0 && errno == EINTR);
>> (gdb) p this->wrMutex
>> $7 = {cs = {__data = {__lock = 1, __count = 0, __owner = 32118, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\001\000\000\000\000\000\000\000v}\000\000\001", '\000' <repeats 26 times>,
>> __align = 1}}
>>
>> 4) Main thread is trying to grab the statsMutex, but it is held by 32118.
>>
>> Thread 42 (Thread 5474):
>> #0 0x0000003de940d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>> #1 0x0000003de9408e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>> #2 0x0000003de9408cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>> #3 0x000000000043bb26 in Lock (Peer=..., opts=0) at ../../src/XrdSys/XrdSysPthread.hh:132
>> #4 XrdLink::Alloc (Peer=..., opts=0) at XrdLink.cc:247
>> #5 0x0000000000443a28 in XrdInet::Accept (this=0x1467bad0, opts=<value optimized out>, timeout=<value optimized out>) at XrdInet.cc:58
>> #6 0x000000000043c4d2 in mainAccept (parg=<value optimized out>) at XrdMain.cc:101
>> #7 0x000000000043c673 in main (argc=9, argv=0x7fff3cd5ffa8) at XrdMain.cc:186
>>
>> So, because one client in Spain has a nasty firewall, the xrootd server in Nebraska stops accepting connections.
>>
>> Ideas?
>>
>> Brian
>>
>> On Mar 3, 2011, at 12:46 PM, Brian Bockelman wrote:
>>
>>> Hi,
>>>
>>> In the same stress test I'm running, I've got about 5 servers that stopped accepting clients. Stack trace below. I accidentally screwed up the process, so no core is available.
>>>
>>> However, I was able to get line numbers:
>>>
>>> (gdb) bt
>>> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>>> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3 0x000000000043bb26 in Lock (Peer=..., opts=0) at ../../src/XrdSys/XrdSysPthread.hh:132
>>> #4 XrdLink::Alloc (Peer=..., opts=0) at XrdLink.cc:247
>>> #5 0x0000000000443a28 in XrdInet::Accept (this=0x14204b10, opts=<value optimized out>, timeout=<value optimized out>) at XrdInet.cc:58
>>> #6 0x000000000043c4d2 in mainAccept (parg=<value optimized out>) at XrdMain.cc:101
>>> #7 0x000000000043c673 in main (argc=9, argv=0x7ffff40c04f8) at XrdMain.cc:186
>>>
>>> The relevant lines from XrdLink.cc are:
>>>
>>>
>>> // Return the link
>>> //
>>> statsMutex.Lock(); <-- line 247
>>> LinkCountTot++;
>>> if (LinkCountMax == LinkCount++) LinkCountMax = LinkCount;
>>> statsMutex.UnLock();
>>>
>>> So - somewhere, the statsMutex got locked and never unlocked.
>>>
>>> Ideas?
>>>
>>> Brian
>>>
>>> Thread 43 (Thread 0x4091b940 (LWP 4286)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
>>> #2 0x0000000000438881 in XrdBuffManager::Reshape() ()
>>> #3 0x0000000000438b99 in XrdReshaper(void*) ()
>>> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 42 (Thread 0x41923940 (LWP 4287)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
>>> #2 0x000000000043fab2 in XrdScheduler::TimeSched() ()
>>> #3 0x000000000043fb19 in XrdStartTSched(void*) ()
>>> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 41 (Thread 0x41a24940 (LWP 4288)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 40 (Thread 0x41b25940 (LWP 4289)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 39 (Thread 0x40168940 (LWP 4290)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
>>> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 38 (Thread 0x41785940 (LWP 4291)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
>>> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 37 (Thread 0x40ed8940 (LWP 4293)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x000000000043cd0d in XrdPollE::Start(XrdSysSemaphore*, int&) ()
>>> #2 0x000000000043c6e8 in XrdStartPolling(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 36 (Thread 0x405cc940 (LWP 4295)):
>>> #0 0x000000355020e1c1 in nanosleep () from /lib64/libpthread.so.0
>>> #1 0x000000000045592a in XrdAccConfig_Refresh(void*) ()
>>> #2 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #3 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #4 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 35 (Thread 0x40a1c940 (LWP 4296)):
>>> #0 0x000000355020d91b in read () from /lib64/libpthread.so.0
>>> #1 0x000000000047cd60 in XrdOucStream::GetLine() ()
>>> #2 0x00000000004479a8 in XrdOfsEvr::recvEvents() ()
>>> #3 0x0000000000447a69 in XrdOfsEvRecv(void*) ()
>>> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 34 (Thread 0x41611940 (LWP 4297)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x0000000000447b2f in XrdOfsEvr::flushEvents() ()
>>> #2 0x0000000000447bd9 in XrdOfsEvFlush(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 33 (Thread 0x40fff940 (LWP 4298)):
>>> #0 0x000000355020dc81 in recv () from /lib64/libpthread.so.0
>>> #1 0x000000000046da37 in XrdCmsFinderTRG::Start() ()
>>> #2 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #3 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #4 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 32 (Thread 0x41e8c940 (LWP 4300)):
>>> #0 0x000000355020db3b in accept () from /lib64/libpthread.so.0
>>> #1 0x0000000000477e24 in XrdNetSocket::Accept(int) ()
>>> #2 0x0000000000422e0d in XrdXrootdAdmin::Start(XrdNetSocket*) ()
>>> #3 0x0000000000423819 in XrdXrootdInitAdmin(void*) ()
>>> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 31 (Thread 0x41f8d940 (LWP 4317)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 30 (Thread 0x4208e940 (LWP 4318)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 29 (Thread 0x4218f940 (LWP 4319)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 28 (Thread 0x42290940 (LWP 4320)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacd24835 in GCTaskManager::get_task(unsigned int) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacd25ae3 in GCTaskThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 27 (Thread 0x41d3a940 (LWP 4321)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7956 in os::PlatformEvent::park(long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7f09b in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaad0ee493 in VMThread::loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaad0ee08e in VMThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 26 (Thread 0x42391940 (LWP 4322)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaad06bf2a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaad069293 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaace01b7e in JVM_MonitorWait () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaab4010c88 in ?? ()
>>> #6 0x00000000151e5800 in ?? ()
>>> #7 0x00002aaaad068eb5 in ObjectSynchronizer::fast_exit(oopDesc*, BasicLock*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #8 0x00002aaab41a81a8 in ?? ()
>>> #9 0x0000000000000000 in ?? ()
>>> Thread 25 (Thread 0x42492940 (LWP 4323)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaad06bf2a in ObjectMonitor::wait(long, bool, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaad069293 in ObjectSynchronizer::wait(Handle, long, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaace01b7e in JVM_MonitorWait () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaab4010c88 in ?? ()
>>> #6 0x00000000151ed000 in ?? ()
>>> #7 0x00000000151ed878 in ?? ()
>>> #8 0x00000000151ed000 in ?? ()
>>> #9 0x0000000042491a08 in ?? ()
>>> #10 0x0000000000000000 in ?? ()
>>> Thread 24 (Thread 0x4045c940 (LWP 4324)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa84a5 in check_pending_signals(bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacfa11c7 in signal_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 23 (Thread 0x40c50940 (LWP 4325)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f5cd in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacc68bf3 in CompileQueue::get() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacc6a781 in CompileBroker::compiler_thread_loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaad0a4969 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #8 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #9 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #10 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 22 (Thread 0x41c26940 (LWP 4326)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f5cd in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacc68bf3 in CompileQueue::get() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaacc6a781 in CompileBroker::compiler_thread_loop() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaad0a4969 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #8 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #9 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #10 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 21 (Thread 0x42593940 (LWP 4327)):
>>> #0 0x000000355020aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7807 in os::PlatformEvent::park() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacf7ef65 in Monitor::IWait(Thread*, long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaacf7f72e in Monitor::wait(bool, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacf28ec2 in LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x00002aaaad09e452 in JavaThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #6 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #7 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #8 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 20 (Thread 0x42694940 (LWP 4328)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x00002aaaacfa7956 in os::PlatformEvent::park(long) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #2 0x00002aaaacfa5b5b in os::sleep(Thread*, long, bool) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #3 0x00002aaaad09d54b in WatcherThread::run() () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #4 0x00002aaaacfa80df in java_start(Thread*) () from /usr/java/latest/jre/lib/amd64/server/libjvm.so
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 19 (Thread 0x42795940 (LWP 4329)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #3 0x00002aaab4157072 in ?? ()
>>> #4 0x00002aaabfdc7750 in ?? ()
>>> #5 0x00002aaab41358e6 in ?? ()
>>> #6 0x00002aab6519ab40 in ?? ()
>>> #7 0x00002aaab7953550 in ?? ()
>>> #8 0x00002aaabfdc7420 in ?? ()
>>> #9 0x00002aaab4136618 in ?? ()
>>> #10 0x0000000042794470 in ?? ()
>>> #11 0x0000000014cd0800 in ?? ()
>>> #12 0x0000000000000001 in ?? ()
>>> #13 0x00002aaab4150634 in ?? ()
>>> #14 0x00002aaabfdc7750 in ?? ()
>>> #15 0x00002aaa00000001 in ?? ()
>>> #16 0x000000000000ea60 in ?? ()
>>> #17 0x00002aaabfdc7888 in ?? ()
>>> #18 0x0000000000000007 in ?? ()
>>> #19 0x00002aaabfdc7750 in ?? ()
>>> #20 0x0000000000000009 in ?? ()
>>> #21 0x00000000427943a8 in ?? ()
>>> #22 0x00002aaabc4c0ea8 in ?? ()
>>> #23 0x00002aaab4161740 in ?? ()
>>> #24 0x00002aab6519ab40 in ?? ()
>>> #25 0x00002aaabfdc7390 in ?? ()
>>> #26 0x00002aaabfdc7720 in ?? ()
>>> #27 0x000000000000ea60 in ?? ()
>>> #28 0x00002aaabfdc7738 in ?? ()
>>> #29 0x00002aaabfdc7390 in ?? ()
>>> #30 0x00002aaabfdc7720 in ?? ()
>>> #31 0x00002aaabfdc7390 in ?? ()
>>> #32 0x00002aaabfdc7738 in ?? ()
>>> #33 0x00002aab651970f8 in ?? ()
>>> #34 0x00002aab6519ab40 in ?? ()
>>> #35 0x00002aab65197028 in ?? ()
>>> #36 0x0000000000000009 in ?? ()
>>> #37 0x0000000000000009 in ?? ()
>>> #38 0x0000000000000009 in ?? ()
>>> #39 0x0000000000000001 in ?? ()
>>> #40 0x0000000000000009 in ?? ()
>>> #41 0x0000000000099473 in ?? ()
>>> #42 0x0000000000000000 in ?? ()
>>> Thread 18 (Thread 0x406cd940 (LWP 802)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 17 (Thread 0x42896940 (LWP 872)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 16 (Thread 0x42997940 (LWP 1695)):
>>> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
>>> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
>>> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
>>> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 15 (Thread 0x41302940 (LWP 1789)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 14 (Thread 0x407ce940 (LWP 1935)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 13 (Thread 0x40b1d940 (LWP 2272)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #3 0x00002aaab4157072 in ?? ()
>>> #4 0x00002aaabfb2eba0 in ?? ()
>>> #5 0x00002aaab41358e6 in ?? ()
>>> #6 0x00002aab649340f0 in ?? ()
>>> #7 0x00002aaab4155e0c in ?? ()
>>> #8 0x00002aaabfb2e870 in ?? ()
>>> #9 0x00002aaab4136618 in ?? ()
>>> #10 0x00002aaac0087b08 in ?? ()
>>> #11 0x00002aaab4136618 in ?? ()
>>> #12 0x0000000000000001 in ?? ()
>>> #13 0x00002aaab4150634 in ?? ()
>>> #14 0x00002aaabfb2eba0 in ?? ()
>>> #15 0x00002aaa00000001 in ?? ()
>>> #16 0x000000000000ea60 in ?? ()
>>> #17 0x00002aaabfb2ecd8 in ?? ()
>>> #18 0x0000000000000007 in ?? ()
>>> #19 0x00002aaabfb2eba0 in ?? ()
>>> #20 0x0000000000000009 in ?? ()
>>> #21 0x00002aaab70c8050 in ?? ()
>>> #22 0x00002aaabc4c0ea8 in ?? ()
>>> #23 0x00002aaab4161740 in ?? ()
>>> #24 0x00002aab649340f0 in ?? ()
>>> #25 0x00002aaabfb2e7e0 in ?? ()
>>> #26 0x00002aaabfb2eb70 in ?? ()
>>> #27 0x000000000000ea60 in ?? ()
>>> #28 0x00002aaabfb2eb88 in ?? ()
>>> #29 0x00002aaabfb2e7e0 in ?? ()
>>> #30 0x00002aaabfb2eb70 in ?? ()
>>> #31 0x00002aaabfb2e7e0 in ?? ()
>>> #32 0x00002aaabfb2eb88 in ?? ()
>>> #33 0x00002aab64928620 in ?? ()
>>> #34 0x00002aab649340f0 in ?? ()
>>> #35 0x00002aab64928550 in ?? ()
>>> #36 0x0000000000000009 in ?? ()
>>> #37 0x0000000000000009 in ?? ()
>>> #38 0x0000000000000009 in ?? ()
>>> #39 0x0000000000000001 in ?? ()
>>> #40 0x0000000000000009 in ?? ()
>>> #41 0x0000000000045367 in ?? ()
>>> #42 0x0000000000000000 in ?? ()
>>> Thread 12 (Thread 0x41100940 (LWP 3448)):
>>> #0 0x000000354f6d4358 in epoll_wait () from /lib64/libc.so.6
>>> #1 0x00002aaaaf0529cd in iepoll () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #2 0x00002aaaaf0528a4 in Java_sun_nio_ch_EPollArrayWrapper_epollWait () from /usr/java/jdk1.6.0_21/jre/lib/amd64/libnio.so
>>> #3 0x00002aaab4157072 in ?? ()
>>> #4 0x00002aaabf9e8448 in ?? ()
>>> #5 0x00002aaab41358e6 in ?? ()
>>> #6 0x00002aab6474b9b8 in ?? ()
>>> #7 0x00002aaab4155e0c in ?? ()
>>> #8 0x00002aaabf9e8118 in ?? ()
>>> #9 0x00002aaab4136618 in ?? ()
>>> #10 0x00002aaabfb2e870 in ?? ()
>>> #11 0x00002aaab4136618 in ?? ()
>>> #12 0x0000000000000001 in ?? ()
>>> #13 0x00002aaab4150634 in ?? ()
>>> #14 0x00002aaabf9e8448 in ?? ()
>>> #15 0x00002aaa00000001 in ?? ()
>>> #16 0x000000000000ea60 in ?? ()
>>> #17 0x00002aaabf9e8580 in ?? ()
>>> #18 0x0000000000000007 in ?? ()
>>> #19 0x00002aaabf9e8448 in ?? ()
>>> #20 0x0000000000000009 in ?? ()
>>> #21 0x00002aaab70c8050 in ?? ()
>>> #22 0x00002aaabc4c0ea8 in ?? ()
>>> #23 0x00002aaab4161740 in ?? ()
>>> #24 0x00002aab6474b9b8 in ?? ()
>>> #25 0x00002aaabf9e8088 in ?? ()
>>> #26 0x00002aaabf9e8418 in ?? ()
>>> #27 0x000000000000ea60 in ?? ()
>>> #28 0x00002aaabf9e8430 in ?? ()
>>> #29 0x00002aaabf9e8088 in ?? ()
>>> #30 0x00002aaabf9e8418 in ?? ()
>>> #31 0x00002aaabf9e8088 in ?? ()
>>> #32 0x00002aaabf9e8430 in ?? ()
>>> #33 0x00002aab646041d8 in ?? ()
>>> #34 0x00002aab6474b9b8 in ?? ()
>>> #35 0x00002aab64604108 in ?? ()
>>> #36 0x0000000000000009 in ?? ()
>>> #37 0x0000000000000009 in ?? ()
>>> #38 0x0000000000000009 in ?? ()
>>> #39 0x0000000000000001 in ?? ()
>>> #40 0x0000000000000009 in ?? ()
>>> #41 0x0000000000043570 in ?? ()
>>> #42 0x0000000000000000 in ?? ()
>>> Thread 11 (Thread 0x41201940 (LWP 3552)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x000000000043e92e in XrdScheduler::Run() ()
>>> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 10 (Thread 0x41504940 (LWP 3862)):
>>> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
>>> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
>>> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
>>> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 9 (Thread 0x42a98940 (LWP 3863)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x000000000043e92e in XrdScheduler::Run() ()
>>> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 8 (Thread 0x42b99940 (LWP 3864)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x000000000043e92e in XrdScheduler::Run() ()
>>> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 7 (Thread 0x42c9a940 (LWP 3865)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x000000000043e92e in XrdScheduler::Run() ()
>>> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 6 (Thread 0x42d9b940 (LWP 3866)):
>>> #0 0x000000354f6ccd83 in writev () from /lib64/libc.so.6
>>> #1 0x000000000043aa06 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 5 (Thread 0x42e9c940 (LWP 3867)):
>>> #0 0x000000355020d89b in write () from /lib64/libpthread.so.0
>>> #1 0x000000000043aac6 in XrdLink::Send(iovec const*, int, int) ()
>>> #2 0x0000000000430004 in XrdXrootdResponse::Send(void*, int) ()
>>> #3 0x0000000000434925 in XrdXrootdProtocol::do_Read() ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 4 (Thread 0x42f9d940 (LWP 3868)):
>>> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>>> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3 0x0000000000439230 in XrdLink::syncStats(int*) ()
>>> #4 0x000000000043a5c1 in XrdLink::Stats(char*, int, int) ()
>>> #5 0x00000000004400f5 in XrdStats::Stats(int) ()
>>> #6 0x00000000004403d4 in XrdStats::Report(char**, int, int) ()
>>> #7 0x0000000000440496 in XrdStatsJob::DoIt() ()
>>> #8 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #9 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #10 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #11 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #12 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 3 (Thread 0x4309e940 (LWP 3869)):
>>> #0 0x000000354f6cb186 in poll () from /lib64/libc.so.6
>>> #1 0x000000000043afe5 in XrdLink::Recv(char*, int, int) ()
>>> #2 0x000000000042df5e in XrdXrootdProtocol::getData(char const*, char*, int) ()
>>> #3 0x000000000042e945 in XrdXrootdProtocol::Process(XrdLink*) ()
>>> #4 0x0000000000439f80 in XrdLink::DoIt() ()
>>> #5 0x000000000043e9a8 in XrdScheduler::Run() ()
>>> #6 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #7 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #8 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #9 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 2 (Thread 0x41403940 (LWP 5571)):
>>> #0 0x000000355020cd01 in sem_wait () from /lib64/libpthread.so.0
>>> #1 0x000000000043e92e in XrdScheduler::Run() ()
>>> #2 0x000000000043eb19 in XrdStartWorking(void*) ()
>>> #3 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 1 (Thread 0x2b151fb5de20 (LWP 4285)):
>>> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>>> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3 0x000000000043bb26 in XrdLink::Alloc(XrdNetPeer&, int) ()
>>> #4 0x0000000000443a28 in XrdInet::Accept(int, int) ()
>>> #5 0x000000000043c4d2 in mainAccept(void*) ()
>>> #6 0x000000000043c673 in main ()
>>> [root@red-gridftp4 ~]# pstack 4285 | head
>>> Thread 43 (Thread 0x4091b940 (LWP 4286)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> #1 0x0000000000481eb2 in XrdSysCondVar::Wait(int) ()
>>> #2 0x0000000000438881 in XrdBuffManager::Reshape() ()
>>> #3 0x0000000000438b99 in XrdReshaper(void*) ()
>>> #4 0x0000000000481d17 in XrdSysThread_Xeq ()
>>> #5 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #6 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 42 (Thread 0x41923940 (LWP 4287)):
>>> #0 0x000000355020b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>>> [root@red-gridftp4 ~]# pstack 4285 | tail
>>> #4 0x000000355020673d in start_thread () from /lib64/libpthread.so.0
>>> #5 0x000000354f6d3f6d in clone () from /lib64/libc.so.6
>>> Thread 1 (Thread 0x2b151fb5de20 (LWP 4285)):
>>> #0 0x000000355020d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1 0x0000003550208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
>>> #2 0x0000003550208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3 0x000000000043bb26 in XrdLink::Alloc(XrdNetPeer&, int) ()
>>> #4 0x0000000000443a28 in XrdInet::Accept(int, int) ()
>>> #5 0x000000000043c4d2 in mainAccept(void*) ()
>>> #6 0x000000000043c673 in main ()
>>>
>>
