Print

Print


Hi there,

since last week I obseve some strange load related problem with the
xrootd cluster at GridKa. During fairly 'normal' load [1] the
redirector reaches its Thread limit [2]. I set it to maximum but the
problems reoccurred several times [3]. It almost seems, that the
scheduler does not flush the threads in time, but I guess might have
more insight on this issue. Well feel free to share any kind of advice
or wisdom :-)
The cluster has 12 servers (2 in staging mode) and all run on 20071101-0808p1.

Florian



[1] # of redirs per 10 minute
http://iktp.tu-dresden.de/~petzold/work/GridKa/xrootd/redirs.png
# of logins per 10 minute
http://iktp.tu-dresden.de/~petzold/work/GridKa/xrootd/logins.png

[2] Log:

080422 00:10:41 27010 odc_send2Man:
mcprod.2729:[log in to unmask] redirected to
f01-010-105.gridka.de:1094 by l01-001-110
path=/store/cfg/2008/03/CfgDB-20080328T123723.root
080422 00:10:42 27010 XrdScheduler: Thread limit has been reached!
080422 00:10:42 27010 odc_send2Man:
mcprod.9859:[log in to unmask] redirected to
f01-010-105.gridka.de:1094 by l01-001-110
path=/store/cfg/2008/03/CfgDB-20080328T123723.root
080422 00:10:42 27010 odc_send2Man:
mcprod.907:[log in to unmask] redirected to
f01-010-105.gridka.de:1094 by l01-001-110
path=/store/cfg/2008/03/CfgDB-20080328T123723.root
080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many
open files in system
080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many
open files in system
080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many
open files in system
080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many
open files in system

[3] Redirector Config:

olb.allow host babar*.gridka.de
olb.allow host f01-*.gridka.de
olb.path s /store
olb.path w /prod
olb.path w /gaffertape
olb.port 3121
olb.sched cpu 100
olb.wait

xrd.sched mint 8 maxt 4095 avlt 512 idle 780

xrootd.export /prod
xrootd.export /store
xrootd.export /gaffertape
xrootd.fslib /home/xrootd/software/current/lib/libXrdOfs.so

odc.trace redirect
#olb.trace all
#oss.trace all
#xrd.trace all
#xrootd.trace all



-- 
------------------------------------------------------------------
Humboldt-Universitšt zu Berlin
Department of Physics
BaBar, Prof. H. Lacker and Prof. M. Kobel
Newtonstr. 15, 12489 Berlin, Germany
Web: slac.stanford.edu/babar
------------------------------------------------------------------