 |
 |
Hi there, since last week I obseve some strange load related problem with the xrootd cluster at GridKa. During fairly 'normal' load [1] the redirector reaches its Thread limit [2]. I set it to maximum but the problems reoccurred several times [3]. It almost seems, that the scheduler does not flush the threads in time, but I guess might have more insight on this issue. Well feel free to share any kind of advice or wisdom :-) The cluster has 12 servers (2 in staging mode) and all run on 20071101-0808p1. Florian [1] # of redirs per 10 minute http://iktp.tu-dresden.de/~petzold/work/GridKa/xrootd/redirs.png # of logins per 10 minute http://iktp.tu-dresden.de/~petzold/work/GridKa/xrootd/logins.png [2] Log: 080422 00:10:41 27010 odc_send2Man: mcprod.2729:[log in to unmask] redirected to f01-010-105.gridka.de:1094 by l01-001-110 path=/store/cfg/2008/03/CfgDB-20080328T123723.root 080422 00:10:42 27010 XrdScheduler: Thread limit has been reached! 080422 00:10:42 27010 odc_send2Man: mcprod.9859:[log in to unmask] redirected to f01-010-105.gridka.de:1094 by l01-001-110 path=/store/cfg/2008/03/CfgDB-20080328T123723.root 080422 00:10:42 27010 odc_send2Man: mcprod.907:[log in to unmask] redirected to f01-010-105.gridka.de:1094 by l01-001-110 path=/store/cfg/2008/03/CfgDB-20080328T123723.root 080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many open files in system 080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many open files in system 080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many open files in system 080422 03:04:21 001 XrdAccept: Unable to perform accept.; too many open files in system [3] Redirector Config: olb.allow host babar*.gridka.de olb.allow host f01-*.gridka.de olb.path s /store olb.path w /prod olb.path w /gaffertape olb.port 3121 olb.sched cpu 100 olb.wait xrd.sched mint 8 maxt 4095 avlt 512 idle 780 xrootd.export /prod xrootd.export /store xrootd.export /gaffertape xrootd.fslib /home/xrootd/software/current/lib/libXrdOfs.so odc.trace redirect #olb.trace all #oss.trace all #xrd.trace all #xrootd.trace all -- ------------------------------------------------------------------ Humboldt-Universität zu Berlin Department of Physics BaBar, Prof. H. Lacker and Prof. M. Kobel Newtonstr. 15, 12489 Berlin, Germany Web: slac.stanford.edu/babar ------------------------------------------------------------------