Hi all,
I got the following on the UNL redirector:
110706 15:20:29 3409 XrdScheduler: Thread limit has been reached!
Sure enough, if you look below, there were a lot of threads - but not that many active users of the cmsd. There were about 500 users connected to the redirector at the time.
Ideas? Should I just bump it and restart?
Brian
[root@xrootd ~]# ps -C cmsd -L | wc -l
2066
[root@xrootd ~]# lsof -p 3409
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cmsd 3409 xrootd cwd DIR 202,3 4096 1335851 /etc/xrootd/MAIN
cmsd 3409 xrootd rtd DIR 202,3 4096 2 /
cmsd 3409 xrootd txt REG 202,3 590160 1265174 /usr/bin/cmsd
cmsd 3409 xrootd mem REG 202,3 139416 484802 /lib64/ld-2.5.so
cmsd 3409 xrootd mem REG 202,3 1718120 484819 /lib64/libc-2.5.so
cmsd 3409 xrootd mem REG 202,3 23360 485018 /lib64/libdl-2.5.so
cmsd 3409 xrootd mem REG 202,3 145824 485021 /lib64/libpthread-2.5.so
cmsd 3409 xrootd mem REG 202,3 615136 484821 /lib64/libm-2.5.so
cmsd 3409 xrootd mem REG 202,3 53448 485022 /lib64/librt-2.5.so
cmsd 3409 xrootd mem REG 202,3 58400 484823 /lib64/libgcc_s-4.1.2-20080825.so.1
cmsd 3409 xrootd mem REG 202,3 114352 484869 /lib64/libnsl-2.5.so
cmsd 3409 xrootd mem REG 202,3 976312 910653 /usr/lib64/libstdc++.so.6.0.8
cmsd 3409 xrootd mem REG 202,3 53880 484850 /lib64/libnss_files-2.5.so
cmsd 3409 xrootd mem REG 202,3 217016 1132028 /var/db/nscd/hosts
cmsd 3409 xrootd 0u CHR 1,3 926 /dev/null
cmsd 3409 xrootd 1u CHR 1,3 926 /dev/null
cmsd 3409 xrootd 2w REG 202,3 3913587 1423082 /var/log/xrootd/MAIN/cmsd.log
cmsd 3409 xrootd 3r 0000 0,11 0 9650817 eventpoll
cmsd 3409 xrootd 4r FIFO 0,6 9650818 pipe
cmsd 3409 xrootd 5w FIFO 0,6 9650818 pipe
cmsd 3409 xrootd 6r 0000 0,11 0 9650819 eventpoll
cmsd 3409 xrootd 7u IPv4 9650827 TCP xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:33154 (CLOSE_WAIT)
cmsd 3409 xrootd 8r FIFO 0,6 9650820 pipe
cmsd 3409 xrootd 9w FIFO 0,6 9650820 pipe
cmsd 3409 xrootd 10r 0000 0,11 0 9650821 eventpoll
cmsd 3409 xrootd 11r FIFO 0,6 9650822 pipe
cmsd 3409 xrootd 12w FIFO 0,6 9650822 pipe
cmsd 3409 xrootd 13u IPv4 9650823 TCP *:mpc-lifenet (LISTEN)
cmsd 3409 xrootd 14u unix 0xffff880054caa980 9650824 /var/run/xrootd/MAIN/.olb/olbd.nimda
cmsd 3409 xrootd 16u IPv4 10319192 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:44711 (CLOSE_WAIT)
cmsd 3409 xrootd 17u IPv4 9844692 TCP xrootd.unl.edu:mpc-lifenet->gridftp-16-23.ultralight.org:45031 (ESTABLISHED)
cmsd 3409 xrootd 18u IPv4 10250440 TCP xrootd.unl.edu:mpc-lifenet->cithep172.ultralight.org:55049 (ESTABLISHED)
cmsd 3409 xrootd 19u IPv4 9900239 TCP xrootd.unl.edu:mpc-lifenet->crabserver.rcac.purdue.edu:58973 (ESTABLISHED)
cmsd 3409 xrootd 20u IPv4 9925712 TCP xrootd.unl.edu:mpc-lifenet->cithep230.ultralight.org:42783 (ESTABLISHED)
cmsd 3409 xrootd 21u IPv4 10319223 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:49612 (CLOSE_WAIT)
cmsd 3409 xrootd 22u IPv4 9887691 TCP xrootd.unl.edu:mpc-lifenet->cmsdbs.rcac.purdue.edu:33708 (ESTABLISHED)
cmsd 3409 xrootd 23u IPv4 9943439 TCP xrootd.unl.edu:mpc-lifenet->cithep251.ultralight.org:58624 (ESTABLISHED)
cmsd 3409 xrootd 24u IPv4 9687047 TCP xrootd.unl.edu:mpc-lifenet->cmssrv32.fnal.gov:47378 (ESTABLISHED)
cmsd 3409 xrootd 25u IPv4 10330951 TCP xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:50468 (ESTABLISHED)
cmsd 3409 xrootd 26u IPv4 9844312 TCP xrootd.unl.edu:mpc-lifenet->cmshdfs02.hep.wisc.edu:36182 (ESTABLISHED)
cmsd 3409 xrootd 27u IPv4 10002012 TCP xrootd.unl.edu:mpc-lifenet->srm.unl.edu:49894 (ESTABLISHED)
cmsd 3409 xrootd 28u IPv4 9900246 TCP xrootd.unl.edu:mpc-lifenet->cithep160.ultralight.org:52890 (ESTABLISHED)
cmsd 3409 xrootd 29u IPv4 10330984 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50437 (CLOSE_WAIT)
cmsd 3409 xrootd 30u IPv4 10331018 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50442 (CLOSE_WAIT)
cmsd 3409 xrootd 31u IPv4 10331054 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52384 (CLOSE_WAIT)
cmsd 3409 xrootd 32u IPv4 10331099 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52388 (CLOSE_WAIT)
cmsd 3409 xrootd 33u IPv4 10331129 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52390 (CLOSE_WAIT)
cmsd 3409 xrootd 34u IPv4 10331162 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52396 (CLOSE_WAIT)
cmsd 3409 xrootd 35u IPv4 10331186 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:34861 (ESTABLISHED)
|