 |
 |
Hi Andy Andrew Hanushevsky wrote: > The message isn't horrible. When xrootd can't create more threads, it > starts > sharing threads. Not ideal, nut workable. Thanks for the information. All my jobs do finish successfully but I wasn't sure of the consequences when more threads couldn't be created. What I need for you to do in > /bin/sh (don't use any other heel) is to give me the output of: > > ulimit -a > sh-2.05a$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 29000 virtual memory (kbytes, -v) unlimited >*and* > ulimit -aH sh-2.05a$ ulimit -aH core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 50000 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 30000 virtual memory (kbytes, -v) unlimited > > That said, in pre 2.4.x kernels, the limits is 256 threads per process. It > is a hard-coded limit and you would need to recompile the kernel to change > it. In 2.4.x kernels, you can change the limit by changing the value in > /proc/sys/kernel/threads-max which sets the maximum total number of > threads. We are using a 2.4 kernel for these machines sh-2.05a$ uname -r 2.4.20-37.7.legacysmp and the number of thread per process is: sh-2.05a$ cat /proc/sys/kernel/threads-max 14336 cheers Manny > ----- Original Message ----- > From: "Emmanuel Olaiya" <[log in to unmask]> > To: <[log in to unmask]> > Sent: Monday, December 13, 2004 2:34 PM > Subject: Problem creating worker threads > > > > Hi Andy > > > > Here is some more information on the problem we are seeing at RAL > > regarding the creation of worker threads. > > > > The number of allowed processes on both the redirector and the server is > > set to 29000 with the max number allowed set to 30000 > > > > /afs/slac.stanford.edu/u/br/olaiya/tmp/proc.txt > > > > Starting 448 jobs (2 jobs on each batch machine), accessing data via > > xrootd we start to see the following output in the server log: > > > > 041205 02:57:58 24692 XrdScheduler: Unable to create worker thread ; > > resource temporarily unavailable > > 041205 02:58:02 24600 XrdScheduler: Unable to create worker thread ; > > resource temporarily unavailable > > > > ( The full log is here: > > /afs/slac.stanford.edu/u/br/olaiya/tmp/xrdlog.20041205 ) > > > > This happens when the number of opens file reaches ~340. At the same > > time when listing the open files we see some of the connections are > > flagged with (CLOSE_WAIT). > > > > The output of lsof on the server can be found here: > > > > /afs/slac.stanford.edu/u/br/olaiya/tmp/server_lsof.txt > > > > Is there some other setting I should tweak in order to allow the > > creation of more worker threads? > > > > cheers > > > > Manny > > > > >