Hi Andy,

On Fri, Sep 10, 2004 at 12:50:21PM -0700, Andrew Hanushevsky wrote:
> >   I assume you mean NFS export, here? (As opposed to xrootd export ;-) It
> > wasn't clear to me that people wanted to do that for /var partitions on the
> > data movers, but it would eliminate Artem's problem.
> Not really, you can just as easily export it via xrood, can't you :-)

  This would be very interesting. As you know I'd like to harvest the log
files from the datamovers for another reasons and I had thought to propose 
an xrootd based export and/or collection mechanism. Perhaps we could put
this together. I'll think about it and make a proposal.

> >   Copying the file to a 2nd server every time one is opened sounds like a
> > lot of extra traffic. At a minimum it would double the I/O out of the data
> > servers (and if the implementation wasn't done in a sensible way it could
> > do more than that).
> It's relative. At some point someone decides the tradeoffs and that's
> somewhat subjective.

  Well, it may be subjective but at least in our case (BaBar's case) doubling
the I/O _always_ would probably be more confusing than simply accepting some
latency to replicate the file from "elsewhere" in the rarer cases where that
needs to be done. (And you need to solve the "replicate the file from elsewhere
problem" anyway to cover the situation where the server has gone down and 
_all_ files that were there are now unavailable, regardless of whether they
were being read at the time.)

> >   If the server has really/truly just gone down (something that shouldn't
> > happen often) it is probably okay to simply delay the client. If it is a load
> > issue, you can probably invent other ways to force the file replication only
> > when it matters (i.e. when some server is over the load threshhold). Anybody
> > for a _2nd_ xrootd/olb system running on the same servers (without the
> > load threshold) dedicated just to file replication?
> I don't think you need two to do the job and would be very confusing if
> you did it that way.

  So how else do I find the file and replicate it from server X to server Y?
xrootd can send the client to server Y, but who tells server Y that it should
copy the file from server X? Is that some "opaque" information that the client
would bring with it?

> >   Doesn't solve the problem, see next point. Also exporting /opt (presumably
> > via NFS, right?) from all data servers is also somewhat ugly.
> No, xrootd.

  Ok, nice.

> >   I really dislike these little "turd" files (feels like VMS). Can't it just
> > make a full copy in memory as it is read in the first time and allow that to
> > be dumped via the adminstrative interface?
> That's somewhat overkill.

  But IMO much better than the stupid turd files that get left behind when the 
server crashes or is shutdown in some non-clean way.

> >   If a data server olbd goes away and then comes back, does the manager
> > olbd clear all entries in its cache from that server olbd?
> Ah, no. Since it's very expensive (latency-wise) to rebuild the cache, the
> manager forces newly rejoined server to refresh the cache for only those
> entries that are actually in the cache at the time the server rejoined.

  I think we said the same thing. Sort-of. By "all entries" I meant those
refering to files on server X, the one that went away. Just so I understand,
let me give an example:

   o (And then there was light)
   o server olbd X connects to the manager olbd
   o various clients come in which are redirected to server X to 
     read files M, N, O and P
   o someone restarts the olbd on server X (but the redirector olbd remains up)
   o the olbd on server X reconnects to the manager olbd

  Now there are two possibilities:

   o the manager olbd _immediately_ tells the server X olbd to check that
     files M, N, O and P are still on server X
   o the manager olbd notes that the server X olbd went AWOL for a bit and
     just marks them as needing to be refreshed in its own cache. The next
     time a client comes in to Open the file, it is as if they had done 
     Open + kXR_Refresh.

  Which of the last two is it? (Or perhaps something else...)

                                   Pete

-------------------------------------------------------------------------
Peter Elmer     E-mail: [log in to unmask]      Phone: +41 (22) 767-4644
Address: CERN Division PPE, Bat. 32 2C-14, CH-1211 Geneva 23, Switzerland
-------------------------------------------------------------------------