Hi Andy
Andrew Hanushevsky wrote:
> The message isn't horrible. When xrootd can't create more threads, it
> starts
> sharing threads. Not ideal, nut workable.
Thanks for the information. All my jobs do finish successfully but I
wasn't sure of the consequences when more threads couldn't be created.
What I need for you to do in
> /bin/sh (don't use any other heel) is to give me the output of:
>
> ulimit -a
>
sh-2.05a$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 29000
virtual memory (kbytes, -v) unlimited
>*and*
> ulimit -aH
sh-2.05a$ ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 30000
virtual memory (kbytes, -v) unlimited
>
> That said, in pre 2.4.x kernels, the limits is 256 threads per process. It
> is a hard-coded limit and you would need to recompile the kernel to change
> it. In 2.4.x kernels, you can change the limit by changing the value in
> /proc/sys/kernel/threads-max which sets the maximum total number of
> threads.
We are using a 2.4 kernel for these machines
sh-2.05a$ uname -r
2.4.20-37.7.legacysmp
and the number of thread per process is:
sh-2.05a$ cat /proc/sys/kernel/threads-max
14336
cheers
Manny
> ----- Original Message -----
> From: "Emmanuel Olaiya" <[log in to unmask]>
> To: <[log in to unmask]>
> Sent: Monday, December 13, 2004 2:34 PM
> Subject: Problem creating worker threads
>
>
> > Hi Andy
> >
> > Here is some more information on the problem we are seeing at RAL
> > regarding the creation of worker threads.
> >
> > The number of allowed processes on both the redirector and the server is
> > set to 29000 with the max number allowed set to 30000
> >
> > /afs/slac.stanford.edu/u/br/olaiya/tmp/proc.txt
> >
> > Starting 448 jobs (2 jobs on each batch machine), accessing data via
> > xrootd we start to see the following output in the server log:
> >
> > 041205 02:57:58 24692 XrdScheduler: Unable to create worker thread ;
> > resource temporarily unavailable
> > 041205 02:58:02 24600 XrdScheduler: Unable to create worker thread ;
> > resource temporarily unavailable
> >
> > ( The full log is here:
> > /afs/slac.stanford.edu/u/br/olaiya/tmp/xrdlog.20041205 )
> >
> > This happens when the number of opens file reaches ~340. At the same
> > time when listing the open files we see some of the connections are
> > flagged with (CLOSE_WAIT).
> >
> > The output of lsof on the server can be found here:
> >
> > /afs/slac.stanford.edu/u/br/olaiya/tmp/server_lsof.txt
> >
> > Is there some other setting I should tweak in order to allow the
> > creation of more worker threads?
> >
> > cheers
> >
> > Manny
> >
> >
>
|