 |
 |
Hi Chris, OK, I'll compare thsi with what is happening with our 2 redirector case. Could you also send log files from each redirector olbd and xrootd; thanks. In the meantime, at least there is a work-around. Andy On Wed, 2 Mar 2005, Brew, CAJ (Chris) wrote: > Hi, > > OK I've done a some more debugging and whilst I haven't caught the > problem on servers that have got the debug option switch on I have got > some more info. > > It's pretty definitely to do with the second load balancer. I ran all > day today with one load balancer without running into the problems but > about 30 mins after I restarted the second load balancer I ran into it > again. Stopping the olbd abd xrootd on the second LB restored access to > the files. > > It's possible I've got this setup misconfigured so the config file from > the LBs is attached below. > > I'll continue to try to get debug output from an LB and a dataserver. > The servers don't crash so I won't be able to get core dumps. > > Yours, > Chris. > > [xrootd107] /opt/xrootd/etc > cat xrootd.cf > # RAL XROOTD base config file > # CAJB 051004 > > # Xrd Configuration > > # xrootd configuration > xrootd.fslib /opt/xrootd/lib/libXrdOfs.so > xrootd.export /store > > #ODC Configuration > odc.manager xrootd107.gridpp.rl.ac.uk 1095 > odc.manager xrootd108.gridpp.rl.ac.uk 1095 > > # OFS Configuration > ofs.redirect remote xrootd107.rl.ac.uk > ofs.redirect remote xrootd108.rl.ac.uk > ofs.redirect target csfnfs35.rl.ac.uk > ofs.redirect target csfnfs41.rl.ac.uk > ofs.redirect target csfnfs45.rl.ac.uk > ofs.redirect target csfnfs46.rl.ac.uk > ofs.redirect target csfnfs47.rl.ac.uk > ofs.redirect target csfnfs48.rl.ac.uk > ofs.redirect target csfnfs49.rl.ac.uk > > #OLB Configuration > olb.port 1095 > olb.subscribe xrootd107.gridpp.rl.ac.uk 1095 > olb.subscribe xrootd108.gridpp.rl.ac.uk 1095 > olb.path r /store > olb.wait > olb.trace debug > > odc.trace debug > ofs.trace debug > > #OSS Configuration > oss.path /store r/o > # Manager config for xrootd redirectors > odc.trace redirect > olb.allow host csfnfs*.rl.ac.uk > olb.allow host csflnx108.rl.ac.uk > olb.allow host xrootd107.gridpp.rl.ac.uk > olb.allow host xrootd108.gridpp.rl.ac.uk > > > -----Original Message----- > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > Sent: 01 March 2005 19:49 > > To: Brew, CAJ (Chris) > > Cc: Olaiya, EO (Emmanuel); [log in to unmask] > > Subject: RE: olbd tracing > > > > Hi Chris, > > > > Yes, especially if it thinks one of the two is not > > particularly responsive > > to its requests. The xrootd redirectors are programmed to be greedy. > > > > Andy > > > > On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > > > > > Hi, > > > > > > The clients should only know about one of the load balancers but the > > > load balancers know about each other from the olbd > > "network" since they > > > share a config file and log into each other. So when the > > client asks the > > > xrootd on the LB server for a file could it be asking the > > olbd on the > > > other LB server to find the file for it. > > > > > > That could explain why what looked to be an intermittant > > problem on the > > > LB server's olbd affected finding files via both LB servers > > in the same > > > way at the same time. > > > > > > Yours, > > > Chris. > > > > > > > -----Original Message----- > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > Sent: 01 March 2005 17:40 > > > > To: Brew, CAJ (Chris) > > > > Cc: Olaiya, EO (Emmanuel); [log in to unmask] > > > > Subject: RE: olbd tracing > > > > > > > > Hi Chris, > > > > > > > > Yes and no. Depends on what client you use. The client > > code has gone > > > > through several changes. Some always ask only one > > redirector, others > > > > switch back and forth. But, in any case, you should have > > > > looked at both > > > > redirector logs. Presumably, the xrootd defaults are > > being used (i.e., > > > > fail-over mode). Take a look at the xrootd logs on the > > > > redirector to see > > > > if anything strange is going on there. > > > > > > > > Andy > > > > > > > > > > > > On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > > > > > > > > > Hmmm... > > > > > > > > > > Actually there are some debug messages in the log file now: > > > > > > > > > > When I just ran another test I got: > > > > > > > > > > 050301 16:46:39 24895 do_Select Lookup delay > > > > xrootd107.gridpp.rl.ac.uk 5 > > > > > 050301 16:46:39 24895 Receive From csfnfs49.rl.ac.uk:1094: > > > > 7@0 have r > > > > > /store/test/csfnfs49.01.root > > > > > 050301 16:46:44 24895 Receive From > > > > xrootd107.gridpp.rl.ac.uk: 35 select > > > > > r /store/test/csfnfs49.01.root > > > > > 050301 16:46:44 24895 do_Select Redirect > > > > xrootd107.gridpp.rl.ac.uk -> > > > > > csfnfs49.rl.ac.uk:1094 for /store/test/csfnfs49.01.root > > > > > > > > > > Is it possible that because there was another load > > balancer in the > > > > > set-up it was asking that to find the files for it? That > > > > other machine > > > > > has now gone down to be reinstalled with SL3 and now we're > > > > getting more > > > > > logging info. > > > > > > > > > > Weird. > > > > > > > > > > Chris. > > > > > > > > > > > -----Original Message----- > > > > > > From: [log in to unmask] > > > > > > [mailto:[log in to unmask]] On Behalf Of > > > > > > Brew, CAJ (Chris) > > > > > > Sent: 01 March 2005 16:13 > > > > > > To: Andrew Hanushevsky; Olaiya, EO (Emmanuel) > > > > > > Cc: [log in to unmask] > > > > > > Subject: RE: olbd tracing > > > > > > > > > > > > cc'd to Manny in case he doesn't catch it on the list. > > > > > > > > > > > > The machine I'm trying to turn the logging on on is > > > > > > xrootd107.gridpp.rl.ac.uk our new master load balancer. > > > > We don't have > > > > > > root access on the box but can restart the deamons and modify > > > > > > the config > > > > > > file. > > > > > > > > > > > > I started the deamon with the -d option by running StopOLB > > > > > > and StartOLB > > > > > > -d rather than using the sudo /sbin/service olbd start|stop > > > > > > we normally > > > > > > do. > > > > > > > > > > > > We haven't had any problems since I reduced the number of > > > > > > xrootd servers > > > > > > in the cluster but I've now only got about 7TB free on the > > > > > > servers left > > > > > > in the cluster so will run out of room in about a week. > > > > > > > > > > > > Yours, > > > > > > Chris. > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > > > Sent: 01 March 2005 15:53 > > > > > > > To: Brew, CAJ (Chris) > > > > > > > Cc: Andrew Hanushevsky; [log in to unmask] > > > > > > > Subject: RE: olbd tracing > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > OK, something seems to be amiss with the overall > > > > > > configuration if even > > > > > > > this doesn't work. Let me get together with Manny and take a > > > > > > > look at what > > > > > > > is actually running and how it is put together. Manny when? > > > > > > > > > > > > > > Andy > > > > > > > > > > > > > > On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I've got: > > > > > > > > > > > > > > > > olb.trace debug > > > > > > > > odc.trace debug > > > > > > > > ofs.trace debug > > > > > > > > > > > > > > > > in my xrootd.cf file and started the olbd with -d on > > > > the LB server > > > > > > > > [xrootd107] /opt/xrootd/etc > ps fwwwU bbdatsrv > > > > > > > > PID TTY STAT TIME COMMAND > > > > > > > > 18801 ? S 0:00 sshd: bbdatsrv@pts/0 > > > > > > > > 18803 pts/0 S 0:00 -bash > > > > > > > > 25484 pts/0 R 0:00 \_ ps fwwwU bbdatsrv > > > > > > > > 24895 pts/0 S 0:00 /opt/xrootd/bin/olbd -d -m -l > > > > > > > > /opt/xrootd/logs/olbdlog -c /opt/xrootd//etc/xrootd.cf > > > > > > > > 23940 pts/0 S 0:00 /opt/xrootd/bin/xrootd -r -l > > > > > > > > /opt/xrootd/logs/xrdlog -c /opt/xrootd/etc/xrootd.cf > > > > > > > > 23975 pts/0 S 0:00 /opt/xrootd/bin/xrootd -r -l > > > > > > > > /opt/xrootd/logs/xrdlog -c /opt/xrootd/etc/xrootd.cf > > > > > > > > > > > > > > > > but am still not getting any debug info on how it's > > > > > > > locating the files: > > > > > > > > > > > > > > > > the old.trace debug on the Data Servers does get me: > > > > > > > > > > > > > > > > 050301 11:45:08 616 Receive From > > xrootd108.gridpp.rl.ac.uk: > > > > > > > 7@0 state > > > > > > > > /store... > > > > > > > > > > > > > > > > when looking for a file. > > > > > > > > > > > > > > > > Anyone know what else I need on the LB server? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Chris. > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > > > > > Sent: 23 February 2005 20:59 > > > > > > > > > To: Brew, CAJ (Chris) > > > > > > > > > Cc: [log in to unmask] > > > > > > > > > Subject: Re: olbd tracing > > > > > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > > > > > That's starting the olbd with the -d option > > (for debugging). > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > From: "Brew, CAJ (Chris)" <[log in to unmask]> > > > > > > > > > To: "Andrew Hanushevsky" <[log in to unmask]> > > > > > > > > > Cc: <[log in to unmask]> > > > > > > > > > Sent: Wednesday, February 23, 2005 11:01 AM > > > > > > > > > Subject: RE: olbd tracing > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > > > > > I don't think the odc.trace redirect is the one > > > > I'm looking > > > > > > > > > for. What's > > > > > > > > > > the directive that puts the "have ?" and > > "have" replies > > > > > > > > > into the olbd > > > > > > > > > > log. > > > > > > > > > > > > > > > > > > > > Once I narrow it down to the manager not > > asking the server > > > > > > > > > or the server > > > > > > > > > > not replying correctly I can turn debug on on > > the relavant > > > > > > > > > machine. I'm > > > > > > > > > > reluctant to turn it on on all machines because > > > > it's a fair > > > > > > > > > time before > > > > > > > > > > the problem manifests itself. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Chris. > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > > > > > > > Sent: 22 February 2005 21:26 > > > > > > > > > > > To: Brew, CAJ (Chris); [log in to unmask] > > > > > > > > > > > Subject: Re: olbd tracing > > > > > > > > > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > > > > > > > > > Try: > > > > > > > > > > > > > > > > > > > > > > odc.trace redirect > > > > > > > > > > > > > > > > > > > > > > for the olb try using the '-d' option; though > > > > you may get > > > > > > > > > > > more information > > > > > > > > > > > than really needed. > > > > > > > > > > > > > > > > > > > > > > Andy > > > > > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > > > From: "Brew, CAJ (Chris)" <[log in to unmask]> > > > > > > > > > > > To: <[log in to unmask]> > > > > > > > > > > > Sent: Tuesday, February 22, 2005 6:53 AM > > > > > > > > > > > Subject: olbd tracing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > What's the trace argument to add to the > > xrootd.cf file > > > > > > > > > to get it to > > > > > > > > > > > > output the queries to locate files to the logs. > > > > > > > > > > > > > > > > > > > > > > > > We're still having problems at RAL with > > > > > > > files/servers not being > > > > > > > > > > > > available via the load balancers when > > they are if you > > > > > > > > > contact them > > > > > > > > > > > > directly. > > > > > > > > > > > > > > > > > > > > > > > > Yours, > > > > > > > > > > > > Chris. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Chris Brew ([log in to unmask]) +44 > > 1235 446326 > > > > > > > > > > > > Particle Physics Department > > > > > > > > > > > > Rutherford Appleton Laboratory > > > > > > > > > > > > Chilton, Didcot. Oxfordshire. > > > > > > > > > > > > OX11 0QX. United Kingdom. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >