Hi Brian,
I took a very deliberate look at the proposal. Well, oddly, it looks like
cmsd is about 95% already there and to get it to 100% shouldn't really
require a protocol change. So, let's take this one point at a time:
1) The cmsd is already capable of handling servers that front a common
filesystem. That's what "role proxy" helps to do. Does it work? Well,
almost. The non-query feature was never implemented because the way people
used "proxy" didn't require it. So, with a bit of tweaking you will be able
to front a shared filesystem with a proxy cluster and that cluster will
never ask whether you have a file or not. It will simply assume you do and
if you don't it won't matter because nobody else does.
2) I appreciate the idea of improving the name caching but the
particular set of algorithms used to cache entries in the current scheme
each run in constant time (that includes cache management which is actually
a much bigger overhead item). I am not at all sure that creating an inode
structure will provide any significant improvement and, in fact, because the
data structure is more complicated, will likely make it less efficient. Let's
say I'm highly skeptical.
Now, if point (2) is actually driven by point (1), which is your desire to
minimize the number of queries to a shared filesystem, then there are
actually better ways of doing this in the redirector itself. For instance,
when dealing with shared filesystem there is no reason why the redirector
can bypass querying servers and should be able to do the query itself. In
fact, we will make this more convenient (it's already possible) largely to
improve the efficiency of firewalled servers relative to meta-managers.
If the above is not the case, then I'm somewhat at a loss about what problem
is being solved here. Surely, the hit rate is much higher than the miss rate
in any properly functioning analysis scenario (in fact, the expected hit
rate should be 100%). Under that assumption, trying to optimize negative
responses is not a particularly effective strategy. On the other hand, the
following sentence in your mail file is incomplete:
"Let's say a client is processing directory /a/{1,2,3,4}. The first call
(prepare or open) will be to /a/1. If a site doesn't have anything in
directory /a/, or if /a/ doesn't exist, the dataserver will respond
with -EEXIST,/a. Then, when the client requests /a/2, any namespaces
without the /a directory will"...
So, perhaps I'm missing something because I don't know what the "will" will
be :-)
Andy
----- Original Message -----
From: "Brian Bockelman" <[log in to unmask]>
To: "xrootd-dev" <[log in to unmask]>
Sent: Wednesday, January 05, 2011 8:31 AM
Subject: Proposed improvements for cmsd protocol
Hi folks,
I have two proposed improvements for the cmsd protocol. Both are designed
to help the scalability of the global namespace project:
1) Namespace IDs: Used by data servers which share a complete namespace
(i.e., serve to load-balance access for an external filesystem). In such a
case, one only needs to query a single server for file information as all
will have the same response. Currently, cmsd will send a query to all
servers. For this case, I propose the data servers will all share a
namespace ID, a unique string identifier given at configuration time. They
can inform their manager/supervisor of this ID, and the manager/supervisor
will use this to minimize repeated queries. Obviously, this is just an
optimization - if two data servers are assigned to different supervisors, it
will not apply.
- Implementation: Add a char* member to CmsLoginData to allow the
namespace to be specified, and pass this information to the Link. This
char* will be treated as the namespace key. Then, keep a separate bitmask
in XrdCmsCluster for namespaces IDs. Upon login, the XrdCmsNode will have a
namespace ID assigned to it (based on the namespace key). For any
broadcast, keep track of the namespace IDs used when looping through
possible nodes, and skip nodes if their namespace ID has already been used.
- The XrdCmsCache object would then operate on namespaces, not nodes.
2) Hierarchical caching: Right now, the cache in a manager/supervisor is a
hash table based keyed by the LFN. Since access patterns (especially in
CMS) tend to correlate with the filesystem hierarchy, I propose to convert
the cache to an inode-like structure. If a disk server does not have a
file, it will still reply with -EEXIST, but also return the most specific
missing directory. The negative response is given to the cache and used to
better filter out namespaces to query.
- Let's say a client is processing directory /a/{1,2,3,4}. The first call
(prepare or open) will be to /a/1. If a site doesn't have anything in
directory /a/, or if /a/ doesn't exist, the dataserver will respond
with -EEXIST,/a. Then, when the client requests /a/2, any namespaces
without the /a directory will
- Implementation: I *think* this can be done mostly by keeping the
negative directory answers in the XrdCmsPList and having Find filter out
namespaces appropriately. XrdCmsPList would be mostly rewritten, but it's
only going to cache responses and never trigger any queries. Hence, it
should be simpler than XrdCmsCache, although it will need a new timer for
expiring entries.
Thoughts? It's not trivial work, but I think the approach is correct.
Brian
