Hi Andy,
On Fri, Sep 10, 2004 at 11:49:15AM -0700, Andrew Hanushevsky wrote:
> > I don't feel strongly about it. Andy should comment, though. I can perhaps
> > see that being able to turn on debugging in-situ without restarting the
> > server might have some value in some particular (theoretical) situation where
> > the server has begun to behave strangely.
> Actually, in this case I do agree with Artem. It would be nice if we could
> turn on/off debugging without restarting the servers. There's nothing in
> the code that prevents it. This may have to be relegated to the
> local admin interface as opposed to the remotely accessible one.
What was the "local admin interface"?
> > > > > Admin should be able to read server's log file(s) remotely, via a call. Log
> > > > > files include main log file, error messages, trace output, pid file.
> > > >
> > > > IIRC, in the original discussion we had about this some of use felt that
> > > > this would be useful, but overkill, since other tools could be used.
> > >
> > > I think overkill is to use 'ssh' and 'more' to examine up to 48 logs files
> > > on 40 hosts. (48 is xrootd, olb, mlog, plog, slog, Slog x 8).
> Yes, overkill, but you can do that today by simply exporting the
> directories that have the log files.
I assume you mean NFS export, here? (As opposed to xrootd export ;-) It
wasn't clear to me that people wanted to do that for /var partitions on the
data movers, but it would eliminate Artem's problem.
> > The whole thing starts to get rather ugly. In the end you will always have
> > some other log file (Ganglia?) that you want to look at which is on the
> > data server itself. This is a general problem so there must be some tool out
> > there to harvest or display log files like this. If not presumably one can
> > probably make something simple as you have undoubtedly already done.
> >
> > (i.e. you can try to convince Andy, but I won't help you do it for this
> > one...)
> Harvesting log files is a good thing and SCS does this for some of their
> own stuff. It's a pain eventhough the script is relatively easy to write.
> The idea is to collect all the log files, insert the host name between the
> timestamp and the text, and then sort the composite to get a full overview
> of events. Sometimes that's useful, mostly not.
Everything was fine until you tried to "composite" them.
> > o Doubling the size of the disk cache and putting more than one copy of
> > every file on different servers in the disk cache (ok, I'm joking).
> Actually, this is a very interesting idea and theoretically significant.
> Suppose we had a dynamic mirroring system. When a server opens a file, it
> copies it to some other server with retention period. That way, when that
> server goes down, any active uses of the file will be redirected to the
> server that has the file (which presumably may or may not choose to
> replicate it as well). This handily solves access to actively used files.
> Unfortuantely, it does nothing for future access to files that reside onoy
> on the server that has gone down.
Well, this is just part of the general issue of how a file is replicated
when the load on one server is deemed too high. Bringing it in from the MSS
or another site might be overkill, but will always work at the expense of
some (probably not excessive) delay for the client.
Copying the file to a 2nd server every time one is opened sounds like a
lot of extra traffic. At a minimum it would double the I/O out of the data
servers (and if the implementation wasn't done in a sensible way it could
do more than that).
If the server has really/truly just gone down (something that shouldn't
happen often) it is probably okay to simply delay the client. If it is a load
issue, you can probably invent other ways to force the file replication only
when it matters (i.e. when some server is over the load threshhold). Anybody
for a _2nd_ xrootd/olb system running on the same servers (without the
load threshold) dedicated just to file replication?
> > > The ultimate goal of remote administaration is not to log into any of kan
> > > servers at all. If you need to log in to check what the config file is,
> > > this complicates your life.
> Again, simply export /opt/xrootd/etc
Doesn't solve the problem, see next point. Also exporting /opt (presumably
via NFS, right?) from all data servers is also somewhat ugly.
> > Simply looking at the config file on disk won't
> > always tell you with what configuration the server was started as it may
> > have been overwritten with a newer version of the config file since the
> > server was stated. You _might_ be able to backtrack through the log files
> > to when the server was started to look at the printout, but if it has
> > been running for many days that might not be so easy. (And in fact the
> > log files could even have been purged.) Andy should comment.
> Quite true. I can modify the start-up script to copy the config file that
> the server was started with under the name, say, "<cfn>.active" to get
> around the moving config file problem. What do people think?
I really dislike these little "turd" files (feels like VMS). Can't it just
make a full copy in memory as it is read in the first time and allow that to
be dumped via the adminstrative interface?
> > > > > Admin should be able to give a signal to dlb to rescan file system for
> > > > > new/gone files.
> > > >
> > > > The olb (once known as the "dlb") doesn't maintain any state on the data
> > > > servers, or have I misunderstood? I'm not sure what this means. The manager
> > > > olbd obviously does have a cache, but as I understand it it also times
> > > > out entries older than 8 hours.
> > >
> > > So the proposal is to make it admin-induced in addition.
> The olbd is very resistant to forced additions with intended use. So,
> that's out of the question. I could see the need for a forced purge of the
> cache (either server-specific or complete). Of course, selective restarts
> would accomplish the same thing at lower cost.
If a data server olbd goes away and then comes back, does the manager
olbd clear all entries in its cache from that server olbd?
> > What exactly would the admin be trying to achieve after having removed
> > a file? If it is simply to save clients some time going through the
> > (ask/be-redirected/not-there/go-back-to-redirector-to-ask-and-refresh)
> > cycle, that could be perhaps be useful, but isn't critical as the client
> > will do the right thing. In practice you probably want some way to something
> > like kXR_refresh without actually opening the file (Kind of like a
> > kXR_forget...) The next client that comes in will then actually trigger
> > the system to find the file. Andy?
> You sort of have that since a server can always send a "file gone"
> notification to the olbd. We have the interface (evenm command line
> script) to do this and it will be (albeit slowly) integrated into the MPS.
> However, Pete is right, all of these types of enhancements simply reduce
> the client overhead but really are not specifically needed for correct
> operation.
Actually, what happens currently when a file is purged off disk because of
the staging? (i.e. because space was needed to stage in some other file) I
guess it is unlikely that this will happen within 8 hours of a file being
used, but theoretically what happens? The next client (assuming that is also
within the 8 hours) is redirected to where it was, doesn't finds it and causes
the refresh? i.e. the purging system doesn't actually propagate any info to the
olbd cache? (Which would be fine, the system is robust without that...)
> > > No. Stopped xrootd needs to tell clients to come back in a certain time
> > > that admin specifies (remotely). For example, if unix-admins need to
> > > reboote a machine to apply a patch, we'd rather have clients wait for 10
> > > minutes rather than redirecting them and restaging a file somewhere else.
> > > So we'd need to tell redirector to hold those clients how need to access
> > > that host to for 10 minutes.
> That actually happens now. When a server goes down, clients will be told
> to wait up to 10 minutes if that server is the only server available to
> serve the requested data. Of course, that's relevant for files we know
> about at the time the request is made. The only way you can prevent
> restaging the data elsewhere in all circumstances is to kill the manager
> olb's during he 10 minute outage (or perhaps have an admin function that
> says to hold all requests for 10 minutes).
This would be an admin function to the xrootd on the redirector, correct?
Would it not be rather "hold all requests which would be redirected to
server X for 10 minutes"?
> > > I am talking about a system that reacts on errors not monitors it. So what
> > > if a filesystem went don't at 2 am - I am less interested to recieve and
> > > alarm, but I'd rather see xrootd reconfigured and restarted to avoid using
> > > that filesystem. Another example: user job chrashes and give message "file
> > > not existent". This leave user wondering why. If xrootd could not only
> > > print this error to client and it's log, but to pass it to some
> > > intellingent error processing system, such system could attempt to find
> > > out why the file is missing and send user more detaind explanation and
> > > suggestions.
> TYpically, all of this is handled by an external monitoring system with
> scriptable alarms. It's standard industry practice now.
Saying that something is "standard industry practice" doesn't convince me
of anything, of course, but obviously I agree with this particular statement...
Pete
-------------------------------------------------------------------------
Peter Elmer E-mail: [log in to unmask] Phone: +41 (22) 767-4644
Address: CERN Division PPE, Bat. 32 2C-14, CH-1211 Geneva 23, Switzerland
-------------------------------------------------------------------------
|
