 |
 |
Hi Brian, Hmmm, are you specifying the xrd.sched maxt directive? If so, shame on you and immediately remove it! If not, is your OS limit set to 500? It shouldn't be, typically it should at least 1K and usually 2k. Is the message coming from the xrootd or the cmsd? It makes a big difference. For the xrootd, the limit can be reached depending on how fast one can turn around a transaction. Internally, it's set to no less than 5 seconds to avoid rescheduling if the client has another request in the queue. For the redirector that may be longer than need be. If it's the cmsd then we need to look where the requests are coming from. This is just a local redirector, yes? Or is this the global one? Andy -----Original Message----- From: Brian Bockelman Sent: Wednesday, July 06, 2011 1:34 PM To: xrootd-dev Subject: Hitting thread limits? Hi all, I got the following on the UNL redirector: 110706 15:20:29 3409 XrdScheduler: Thread limit has been reached! Sure enough, if you look below, there were a lot of threads - but not that many active users of the cmsd. There were about 500 users connected to the redirector at the time. Ideas? Should I just bump it and restart? Brian [root@xrootd ~]# ps -C cmsd -L | wc -l 2066 [root@xrootd ~]# lsof -p 3409 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME cmsd 3409 xrootd cwd DIR 202,3 4096 1335851 /etc/xrootd/MAIN cmsd 3409 xrootd rtd DIR 202,3 4096 2 / cmsd 3409 xrootd txt REG 202,3 590160 1265174 /usr/bin/cmsd cmsd 3409 xrootd mem REG 202,3 139416 484802 /lib64/ld-2.5.so cmsd 3409 xrootd mem REG 202,3 1718120 484819 /lib64/libc-2.5.so cmsd 3409 xrootd mem REG 202,3 23360 485018 /lib64/libdl-2.5.so cmsd 3409 xrootd mem REG 202,3 145824 485021 /lib64/libpthread-2.5.so cmsd 3409 xrootd mem REG 202,3 615136 484821 /lib64/libm-2.5.so cmsd 3409 xrootd mem REG 202,3 53448 485022 /lib64/librt-2.5.so cmsd 3409 xrootd mem REG 202,3 58400 484823 /lib64/libgcc_s-4.1.2-20080825.so.1 cmsd 3409 xrootd mem REG 202,3 114352 484869 /lib64/libnsl-2.5.so cmsd 3409 xrootd mem REG 202,3 976312 910653 /usr/lib64/libstdc++.so.6.0.8 cmsd 3409 xrootd mem REG 202,3 53880 484850 /lib64/libnss_files-2.5.so cmsd 3409 xrootd mem REG 202,3 217016 1132028 /var/db/nscd/hosts cmsd 3409 xrootd 0u CHR 1,3 926 /dev/null cmsd 3409 xrootd 1u CHR 1,3 926 /dev/null cmsd 3409 xrootd 2w REG 202,3 3913587 1423082 /var/log/xrootd/MAIN/cmsd.log cmsd 3409 xrootd 3r 0000 0,11 0 9650817 eventpoll cmsd 3409 xrootd 4r FIFO 0,6 9650818 pipe cmsd 3409 xrootd 5w FIFO 0,6 9650818 pipe cmsd 3409 xrootd 6r 0000 0,11 0 9650819 eventpoll cmsd 3409 xrootd 7u IPv4 9650827 TCP xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:33154 (CLOSE_WAIT) cmsd 3409 xrootd 8r FIFO 0,6 9650820 pipe cmsd 3409 xrootd 9w FIFO 0,6 9650820 pipe cmsd 3409 xrootd 10r 0000 0,11 0 9650821 eventpoll cmsd 3409 xrootd 11r FIFO 0,6 9650822 pipe cmsd 3409 xrootd 12w FIFO 0,6 9650822 pipe cmsd 3409 xrootd 13u IPv4 9650823 TCP *:mpc-lifenet (LISTEN) cmsd 3409 xrootd 14u unix 0xffff880054caa980 9650824 /var/run/xrootd/MAIN/.olb/olbd.nimda cmsd 3409 xrootd 16u IPv4 10319192 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:44711 (CLOSE_WAIT) cmsd 3409 xrootd 17u IPv4 9844692 TCP xrootd.unl.edu:mpc-lifenet->gridftp-16-23.ultralight.org:45031 (ESTABLISHED) cmsd 3409 xrootd 18u IPv4 10250440 TCP xrootd.unl.edu:mpc-lifenet->cithep172.ultralight.org:55049 (ESTABLISHED) cmsd 3409 xrootd 19u IPv4 9900239 TCP xrootd.unl.edu:mpc-lifenet->crabserver.rcac.purdue.edu:58973 (ESTABLISHED) cmsd 3409 xrootd 20u IPv4 9925712 TCP xrootd.unl.edu:mpc-lifenet->cithep230.ultralight.org:42783 (ESTABLISHED) cmsd 3409 xrootd 21u IPv4 10319223 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:49612 (CLOSE_WAIT) cmsd 3409 xrootd 22u IPv4 9887691 TCP xrootd.unl.edu:mpc-lifenet->cmsdbs.rcac.purdue.edu:33708 (ESTABLISHED) cmsd 3409 xrootd 23u IPv4 9943439 TCP xrootd.unl.edu:mpc-lifenet->cithep251.ultralight.org:58624 (ESTABLISHED) cmsd 3409 xrootd 24u IPv4 9687047 TCP xrootd.unl.edu:mpc-lifenet->cmssrv32.fnal.gov:47378 (ESTABLISHED) cmsd 3409 xrootd 25u IPv4 10330951 TCP xrootd.unl.edu:mpc-lifenet->xrootd.unl.edu:50468 (ESTABLISHED) cmsd 3409 xrootd 26u IPv4 9844312 TCP xrootd.unl.edu:mpc-lifenet->cmshdfs02.hep.wisc.edu:36182 (ESTABLISHED) cmsd 3409 xrootd 27u IPv4 10002012 TCP xrootd.unl.edu:mpc-lifenet->srm.unl.edu:49894 (ESTABLISHED) cmsd 3409 xrootd 28u IPv4 9900246 TCP xrootd.unl.edu:mpc-lifenet->cithep160.ultralight.org:52890 (ESTABLISHED) cmsd 3409 xrootd 29u IPv4 10330984 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50437 (CLOSE_WAIT) cmsd 3409 xrootd 30u IPv4 10331018 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:50442 (CLOSE_WAIT) cmsd 3409 xrootd 31u IPv4 10331054 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52384 (CLOSE_WAIT) cmsd 3409 xrootd 32u IPv4 10331099 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52388 (CLOSE_WAIT) cmsd 3409 xrootd 33u IPv4 10331129 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52390 (CLOSE_WAIT) cmsd 3409 xrootd 34u IPv4 10331162 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:52396 (CLOSE_WAIT) cmsd 3409 xrootd 35u IPv4 10331186 TCP xrootd.unl.edu:mpc-lifenet->xrootd.t2.ucsd.edu:34861 (ESTABLISHED)