Hi Brian,

Hmmm, are you specifying the xrd.sched maxt directive? If so, shame on you
and immediately remove it!

If not, is your OS limit set to 500? It shouldn't be, typically it should at
least 1K and usually 2k. Is the message coming from the xrootd or the cmsd?
It makes a big difference. For the xrootd, the limit can be reached
depending on how fast one can turn around a transaction. Internally, it's
set to no less than 5 seconds to avoid rescheduling if the client has
another request in the queue. For the redirector that may be longer than
need be. If it's the cmsd then we need to look where the requests are coming
from. This is just a local redirector, yes? Or is this the global one?

Andy

-----Original Message-----
From: Brian Bockelman
Sent: Wednesday, July 06, 2011 1:34 PM
To: xrootd-dev
Subject: Hitting thread limits?


Hi all,

I got the following on the UNL redirector:

110706 15:20:29 3409 XrdScheduler: Thread limit has been reached!

Sure enough, if you look below, there were a lot of threads - but not that
many active users of the cmsd. There were about 500 users connected to the
redirector at the time.

Ideas? Should I just bump it and restart?

Brian

[root@xrootd ~]# ps -C cmsd -L | wc -l
2066
[root@xrootd ~]# lsof -p 3409
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cmsd 3409 xrootd cwd DIR 202,3 4096 1335851
/etc/xrootd/MAIN
cmsd 3409 xrootd rtd DIR 202,3 4096 2 /
cmsd 3409 xrootd txt REG 202,3 590160 1265174
/usr/bin/cmsd
cmsd 3409 xrootd mem REG 202,3 139416 484802
/lib64/ld-2.5.so
cmsd 3409 xrootd mem REG 202,3 1718120 484819
/lib64/libc-2.5.so
cmsd 3409 xrootd mem REG 202,3 23360 485018
/lib64/libdl-2.5.so
cmsd 3409 xrootd mem REG 202,3 145824 485021
/lib64/libpthread-2.5.so
cmsd 3409 xrootd mem REG 202,3 615136 484821
/lib64/libm-2.5.so
cmsd 3409 xrootd mem REG 202,3 53448 485022
/lib64/librt-2.5.so
cmsd 3409 xrootd mem REG 202,3 58400 484823
/lib64/libgcc_s-4.1.2-20080825.so.1
cmsd 3409 xrootd mem REG 202,3 114352 484869
/lib64/libnsl-2.5.so
cmsd 3409 xrootd mem REG 202,3 976312 910653
/usr/lib64/libstdc++.so.6.0.8
cmsd 3409 xrootd mem REG 202,3 53880 484850
/lib64/libnss_files-2.5.so
cmsd 3409 xrootd mem REG 202,3 217016 1132028
/var/db/nscd/hosts
cmsd 3409 xrootd 0u CHR 1,3 926 /dev/null
cmsd 3409 xrootd 1u CHR 1,3 926 /dev/null
cmsd 3409 xrootd 2w REG 202,3 3913587 1423082
/var/log/xrootd/MAIN/cmsd.log
cmsd 3409 xrootd 3r 0000 0,11 0 9650817 eventpoll
cmsd 3409 xrootd 4r FIFO 0,6 9650818 pipe
cmsd 3409 xrootd 5w FIFO 0,6 9650818 pipe
cmsd 3409 xrootd 6r 0000 0,11 0 9650819 eventpoll
cmsd 3409 xrootd 7u IPv4 9650827 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:33154 (CLOSE_WAIT)
cmsd 3409 xrootd 8r FIFO 0,6 9650820 pipe
cmsd 3409 xrootd 9w FIFO 0,6 9650820 pipe
cmsd 3409 xrootd 10r 0000 0,11 0 9650821 eventpoll
cmsd 3409 xrootd 11r FIFO 0,6 9650822 pipe
cmsd 3409 xrootd 12w FIFO 0,6 9650822 pipe
cmsd 3409 xrootd 13u IPv4 9650823 TCP
*:mpc-lifenet (LISTEN)
cmsd 3409 xrootd 14u unix 0xffff880054caa980 9650824
/var/run/xrootd/MAIN/.olb/olbd.nimda
cmsd 3409 xrootd 16u IPv4 10319192 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:44711 (CLOSE_WAIT)
cmsd 3409 xrootd 17u IPv4 9844692 TCP
xrootd.unl.edu:mpc-lifenet->gridftp-16-23.ultralight.org:45031 (ESTABLISHED)
cmsd 3409 xrootd 18u IPv4 10250440 TCP
xrootd.unl.edu:mpc-lifenet->cithep172.ultralight.org:55049 (ESTABLISHED)
cmsd 3409 xrootd 19u IPv4 9900239 TCP
xrootd.unl.edu:mpc-lifenet->crabserver.rcac.purdue.edu:58973 (ESTABLISHED)
cmsd 3409 xrootd 20u IPv4 9925712 TCP
xrootd.unl.edu:mpc-lifenet->cithep230.ultralight.org:42783 (ESTABLISHED)
cmsd 3409 xrootd 21u IPv4 10319223 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:49612 (CLOSE_WAIT)
cmsd 3409 xrootd 22u IPv4 9887691 TCP
xrootd.unl.edu:mpc-lifenet->cmsdbs.rcac.purdue.edu:33708 (ESTABLISHED)
cmsd 3409 xrootd 23u IPv4 9943439 TCP
xrootd.unl.edu:mpc-lifenet->cithep251.ultralight.org:58624 (ESTABLISHED)
cmsd 3409 xrootd 24u IPv4 9687047 TCP
xrootd.unl.edu:mpc-lifenet->cmssrv32.fnal.gov:47378 (ESTABLISHED)
cmsd 3409 xrootd 25u IPv4 10330951 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:50468 (ESTABLISHED)
cmsd 3409 xrootd 26u IPv4 9844312 TCP
xrootd.unl.edu:mpc-lifenet->cmshdfs02.hep.wisc.edu:36182 (ESTABLISHED)
cmsd 3409 xrootd 27u IPv4 10002012 TCP
xrootd.unl.edu:mpc-lifenet->srm.unl.edu:49894 (ESTABLISHED)
cmsd 3409 xrootd 28u IPv4 9900246 TCP
xrootd.unl.edu:mpc-lifenet->cithep160.ultralight.org:52890 (ESTABLISHED)
cmsd 3409 xrootd 29u IPv4 10330984 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50437 (CLOSE_WAIT)
cmsd 3409 xrootd 30u IPv4 10331018 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50442 (CLOSE_WAIT)
cmsd 3409 xrootd 31u IPv4 10331054 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52384 (CLOSE_WAIT)
cmsd 3409 xrootd 32u IPv4 10331099 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52388 (CLOSE_WAIT)
cmsd 3409 xrootd 33u IPv4 10331129 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52390 (CLOSE_WAIT)
cmsd 3409 xrootd 34u IPv4 10331162 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52396 (CLOSE_WAIT)
cmsd 3409 xrootd 35u IPv4 10331186 TCP
xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:34861 (ESTABLISHED)