Hi Chris, Yes, especially if it thinks one of the two is not particularly responsive to its requests. The xrootd redirectors are programmed to be greedy. Andy On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > Hi, > > The clients should only know about one of the load balancers but the > load balancers know about each other from the olbd "network" since they > share a config file and log into each other. So when the client asks the > xrootd on the LB server for a file could it be asking the olbd on the > other LB server to find the file for it. > > That could explain why what looked to be an intermittant problem on the > LB server's olbd affected finding files via both LB servers in the same > way at the same time. > > Yours, > Chris. > > > -----Original Message----- > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > Sent: 01 March 2005 17:40 > > To: Brew, CAJ (Chris) > > Cc: Olaiya, EO (Emmanuel); [log in to unmask] > > Subject: RE: olbd tracing > > > > Hi Chris, > > > > Yes and no. Depends on what client you use. The client code has gone > > through several changes. Some always ask only one redirector, others > > switch back and forth. But, in any case, you should have > > looked at both > > redirector logs. Presumably, the xrootd defaults are being used (i.e., > > fail-over mode). Take a look at the xrootd logs on the > > redirector to see > > if anything strange is going on there. > > > > Andy > > > > > > On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > > > > > Hmmm... > > > > > > Actually there are some debug messages in the log file now: > > > > > > When I just ran another test I got: > > > > > > 050301 16:46:39 24895 do_Select Lookup delay > > xrootd107.gridpp.rl.ac.uk 5 > > > 050301 16:46:39 24895 Receive From csfnfs49.rl.ac.uk:1094: > > 7@0 have r > > > /store/test/csfnfs49.01.root > > > 050301 16:46:44 24895 Receive From > > xrootd107.gridpp.rl.ac.uk: 35 select > > > r /store/test/csfnfs49.01.root > > > 050301 16:46:44 24895 do_Select Redirect > > xrootd107.gridpp.rl.ac.uk -> > > > csfnfs49.rl.ac.uk:1094 for /store/test/csfnfs49.01.root > > > > > > Is it possible that because there was another load balancer in the > > > set-up it was asking that to find the files for it? That > > other machine > > > has now gone down to be reinstalled with SL3 and now we're > > getting more > > > logging info. > > > > > > Weird. > > > > > > Chris. > > > > > > > -----Original Message----- > > > > From: [log in to unmask] > > > > [mailto:[log in to unmask]] On Behalf Of > > > > Brew, CAJ (Chris) > > > > Sent: 01 March 2005 16:13 > > > > To: Andrew Hanushevsky; Olaiya, EO (Emmanuel) > > > > Cc: [log in to unmask] > > > > Subject: RE: olbd tracing > > > > > > > > cc'd to Manny in case he doesn't catch it on the list. > > > > > > > > The machine I'm trying to turn the logging on on is > > > > xrootd107.gridpp.rl.ac.uk our new master load balancer. > > We don't have > > > > root access on the box but can restart the deamons and modify > > > > the config > > > > file. > > > > > > > > I started the deamon with the -d option by running StopOLB > > > > and StartOLB > > > > -d rather than using the sudo /sbin/service olbd start|stop > > > > we normally > > > > do. > > > > > > > > We haven't had any problems since I reduced the number of > > > > xrootd servers > > > > in the cluster but I've now only got about 7TB free on the > > > > servers left > > > > in the cluster so will run out of room in about a week. > > > > > > > > Yours, > > > > Chris. > > > > > > > > > -----Original Message----- > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > Sent: 01 March 2005 15:53 > > > > > To: Brew, CAJ (Chris) > > > > > Cc: Andrew Hanushevsky; [log in to unmask] > > > > > Subject: RE: olbd tracing > > > > > > > > > > Hi Chris, > > > > > > > > > > OK, something seems to be amiss with the overall > > > > configuration if even > > > > > this doesn't work. Let me get together with Manny and take a > > > > > look at what > > > > > is actually running and how it is put together. Manny when? > > > > > > > > > > Andy > > > > > > > > > > On Tue, 1 Mar 2005, Brew, CAJ (Chris) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I've got: > > > > > > > > > > > > olb.trace debug > > > > > > odc.trace debug > > > > > > ofs.trace debug > > > > > > > > > > > > in my xrootd.cf file and started the olbd with -d on > > the LB server > > > > > > [xrootd107] /opt/xrootd/etc > ps fwwwU bbdatsrv > > > > > > PID TTY STAT TIME COMMAND > > > > > > 18801 ? S 0:00 sshd: bbdatsrv@pts/0 > > > > > > 18803 pts/0 S 0:00 -bash > > > > > > 25484 pts/0 R 0:00 \_ ps fwwwU bbdatsrv > > > > > > 24895 pts/0 S 0:00 /opt/xrootd/bin/olbd -d -m -l > > > > > > /opt/xrootd/logs/olbdlog -c /opt/xrootd//etc/xrootd.cf > > > > > > 23940 pts/0 S 0:00 /opt/xrootd/bin/xrootd -r -l > > > > > > /opt/xrootd/logs/xrdlog -c /opt/xrootd/etc/xrootd.cf > > > > > > 23975 pts/0 S 0:00 /opt/xrootd/bin/xrootd -r -l > > > > > > /opt/xrootd/logs/xrdlog -c /opt/xrootd/etc/xrootd.cf > > > > > > > > > > > > but am still not getting any debug info on how it's > > > > > locating the files: > > > > > > > > > > > > the old.trace debug on the Data Servers does get me: > > > > > > > > > > > > 050301 11:45:08 616 Receive From xrootd108.gridpp.rl.ac.uk: > > > > > 7@0 state > > > > > > /store... > > > > > > > > > > > > when looking for a file. > > > > > > > > > > > > Anyone know what else I need on the LB server? > > > > > > > > > > > > Thanks, > > > > > > Chris. > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > > > Sent: 23 February 2005 20:59 > > > > > > > To: Brew, CAJ (Chris) > > > > > > > Cc: [log in to unmask] > > > > > > > Subject: Re: olbd tracing > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > That's starting the olbd with the -d option (for debugging). > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > From: "Brew, CAJ (Chris)" <[log in to unmask]> > > > > > > > To: "Andrew Hanushevsky" <[log in to unmask]> > > > > > > > Cc: <[log in to unmask]> > > > > > > > Sent: Wednesday, February 23, 2005 11:01 AM > > > > > > > Subject: RE: olbd tracing > > > > > > > > > > > > > > > > > > > > > > Hi Andy, > > > > > > > > > > > > > > > > I don't think the odc.trace redirect is the one > > I'm looking > > > > > > > for. What's > > > > > > > > the directive that puts the "have ?" and "have" replies > > > > > > > into the olbd > > > > > > > > log. > > > > > > > > > > > > > > > > Once I narrow it down to the manager not asking the server > > > > > > > or the server > > > > > > > > not replying correctly I can turn debug on on the relavant > > > > > > > machine. I'm > > > > > > > > reluctant to turn it on on all machines because > > it's a fair > > > > > > > time before > > > > > > > > the problem manifests itself. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Chris. > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Andrew Hanushevsky [mailto:[log in to unmask]] > > > > > > > > > Sent: 22 February 2005 21:26 > > > > > > > > > To: Brew, CAJ (Chris); [log in to unmask] > > > > > > > > > Subject: Re: olbd tracing > > > > > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > > > > > Try: > > > > > > > > > > > > > > > > > > odc.trace redirect > > > > > > > > > > > > > > > > > > for the olb try using the '-d' option; though > > you may get > > > > > > > > > more information > > > > > > > > > than really needed. > > > > > > > > > > > > > > > > > > Andy > > > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > > > > > > From: "Brew, CAJ (Chris)" <[log in to unmask]> > > > > > > > > > To: <[log in to unmask]> > > > > > > > > > Sent: Tuesday, February 22, 2005 6:53 AM > > > > > > > > > Subject: olbd tracing > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > What's the trace argument to add to the xrootd.cf file > > > > > > > to get it to > > > > > > > > > > output the queries to locate files to the logs. > > > > > > > > > > > > > > > > > > > > We're still having problems at RAL with > > > > > files/servers not being > > > > > > > > > > available via the load balancers when they are if you > > > > > > > contact them > > > > > > > > > > directly. > > > > > > > > > > > > > > > > > > > > Yours, > > > > > > > > > > Chris. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Chris Brew ([log in to unmask]) +44 1235 446326 > > > > > > > > > > Particle Physics Department > > > > > > > > > > Rutherford Appleton Laboratory > > > > > > > > > > Chilton, Didcot. Oxfordshire. > > > > > > > > > > OX11 0QX. United Kingdom. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >