 |
 |
Hi Andy, I had "cms.trace all" all along. This is the extract of redirects: http://uaf-2.t2.ucsd.edu/~matevz/tmp/cmsd-redirect.txt The full log: http://uaf-2.t2.ucsd.edu/~matevz/tmp/cmsd.log And a sortable table of a set of ~200 files opened with 1 second interval: http://uaf-2.t2.ucsd.edu/~matevz/tmp/ucsd-openfiles.html - you can sort it by open time (similar to redirect extract); - or by server name to see the distribution over servers. Our servers are uaf-[3-9], cabinet-8-8-[0-8], cabinet-8-8-[10-13]. You'll see that cabinet 0, 2, 3, 7, 8 and 10 do not get selected at all in this 200 file test and that uaf-4, 5 and 9 are only selected 2 or 3 times. I checked there is no weirdness on xrootd / cmsd logs on the under provisioned nodes (and that I can talk to them directly). Ah, just noticed ... the cabinet nodes that don't get selected do have a higher load & cpu usage and the ones that do are not doing anything (which is really unusual, that's why I didn't even check it at first). So my cms.sched settings seem to get ignored! The full config, redirector is xrootd.t2.ucsd.edu: http://uaf-2.t2.ucsd.edu/~matevz/tmp/xrootd.cfg Matevz On 02/27/14 01:05, Andrew Hanushevsky wrote: > Hi Matevz, > > The only way to find out is to turn on redirect debugging in the cmsd for a > while and see what the decisions were. We can go from there once we have a > timeline. > > Andy > > On Wed, 26 Feb 2014, Matevz Tadel wrote: > >> On 02/26/14 09:22, Matevz Tadel wrote: >>> Hi, >>> >>> We have ~20 of xrootd servers at UCSD, all of them do something else, too, and >>> are thus under different load. This led to practically all requests going to a >>> few servers only so I set cms.sched to do round-robin. But this does't help >>> much, the open requests are still mostly sent to the same few servers. >>> >>> Could it be that "cms.dfs lookup distrib" causes the redirector to send the >>> client to the "fastest to respond" server instead of decoupling verify and >>> redirect steps? >> >> OK, that wasn't it ... I got hdfs configured on our redirector and tried >> lookup central but it didn't change anything. >> >> What could cause the redirector to only redirect to a few servers? I have this >> now ... so it should be pure round-robin, right? >> cms.sched cpu 0 io 0 mem 0 pag 0 runq 0 space 0 fuzz 100 refreset 3600 >> >> >> Matevz >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1