Hi all,

In preparation for the meeting, I'd like to mention that we have had a
successful clustering of 175 servers. We will be trying for 280 servers
today as well as setting p some stressful scenarios. Below is the first
draft of the documentation that will be added to he olb configuration


Configuring more than 64 data servers

1) Choose the port numbers you wish to use for xrootd and olbd.
Typically, for xrootd port 1094 is used. For olbd port 1213 is used.

1) Choose the number of manager nodes you wish to run.
You must configure at least one manager node. The manager is the first
point of contact for a client and is also the cluster leader. A manager
should run on a dedicated machine of modest power (e.g., 512MB RAM, 800MHZ
clock speed, 100Mb ethernet).

A manager node consists of
a) an xrootd started with the -r option or configured with the
"ofs.redirect remote" directive.
b) an olbd started with the -m option.
You may configure more than one manager and either run them in fail-over
mode (the default) or in load balancing mode where each manager shares
part of the client load (see the ??? directive). Each manager xrootd-olbd
pair must run on a separate machine.

A typical manager configuration file useful for an xrootd and olbd would
look like:

2) Choose the number of supervisor nodes you need.
A supervisor node acts as a local manager for a group of 64 nodes. These
nodes may be data servers or supervisors. A supervisor node consists of
a) an xrootd started with the -r and -t option or configured with the
"ofs.redirect remote" and "ofs.redirect target" directives. Additionally,
the xrootd should be started with the -a option.
b) an olbd started with the -m and -s options.

You only need to configure supervisor nodes if you will be running more
than 64 data servers. The number of supervisor nodes is based on the
number of available manager plus supervisor slots. A recursive formula
must be used to calculate the minimum number. Since there is nothing wrong
in starting a few more supervisors than actually are needed, a simplified
formula can be used.

Conservatively, you will need one supervisor node for each group of 64
data servers. For instance, if you plan to run 500 data servers you will
need the upper limit of 500/64 supervisors (i.e., 8). The computed number
must then be multiplied by the number of managers you will run (e.g., 16
supervisors if you are running two managers). The reason is that
supervisors establish the same number of different paths to data servers
as there are possible client paths to managers.

Each supervisor nodes can run on a data server node. If you wish to share
resources in this way, choose data server nodes that will be as lightly
loaded as possible. The performance requirements for a supervisor node are
the same as a manager node.

4) Configure the data server nodes.
A data server node is delivers actual data to clients. It consists of
a) an xrootd started with the -t option or configured with the
"ofs.redirect target" directive. Additionally, the xrootd should be
started with the -a option.
b) an olbd started with the -s option.

Configure as many data server nodes as you need. Keep in mind that at
least one additional supervisor node is need for each group of 64 data

The performance requirements are determined by the performance needs of
clients. The server should have enough disk space, adequate network
bandwidth (e.g., Gb ethernet), and significant cpu and i/o resources. If
you wish to use memory mapped files, then the node should have a
commensurate amount of real memory.

Frequently asked questions:

Does start-up order matter?

Generally, it does not matter in which order nodes are started. For the
efficiency minded, starting supervisor nodes ahead of data server nodes
allows the system to converge on a stable configuration faster.

How long will it take for the system converge?

This depends on how many servers are in the configuration. Generally, it
takes approximately 1 to 13 seconds for a server to find its place in the
hierarchy. However, the process is parallel across all of the servers. So,
the system should converge in less than 30 seconds for a configuration of
about 1,000 servers.

What happens if I have too few supervisors?

If there are not enough supervisors relative to the number of data
servers, one ore more data servers will be orphaned. If you suspect this,
check the managerís log. It will contain warnings about orphaned data

What happens if I have more supervisor nodes than I need?

Since the system tries to evenly distribute data servers across all
available supervisors, excess supervisors are used to further reduce the
load on supervisor nodes.

Why do I need multiply the number of needed supervisors by the number of

Replicating managers indicates that you wish to have redundant paths to
server nodes. This can only be accomplished if each manager has as many
paths to a data server node. as clients have to a manager node. This
necessitates replicating supervisor nodes by the number of accessible
manager nodes. While system will work if you violate this principal, it
will be in a perpetual path discovery state and perform poorly.

Can I run all the supervisors on a single node?

Yes, but you will need to manually manage the olb administration path and
log files. Each supervisor olbd must have a unique administration path
defined by the olb.adminpath directive. If you are not sharing the
configuration file with the its xrootd counterpart, the path must be
declared to the corresponding xrootd using the odc.olbpath directive.
These requirements mean that each server pair will likely need its own
configuration file. Additionally, each server (olbd and xrootd) must also
be given a separate log file using the -l command line option. Finally,
each xrootd must be started with the -a option to allow for arbitrary port
selection and the port must not be specified using the xrd.port directive.
You should realize that running all of the supervisors on a single node
creates a large single point of failure.

How do I run a data server and a supervisor on the same node?

Use the provided StartOLB and StartXRD scripts. For a supervisor olbd,
specify the -m and -s options. For the supervisorís xrootd, specify the -r
and -t options. For a data server olbd just specify the -s option. For its
xrootd counterpart specify the -t option. The scripts will make sure that
unique log files are established for each server pair and the no port
conflicts will occur by automatically adding the -a command line option
for the xrootd.

What does the -a xrootd command line option actually do?

The -a option enables anarchist mode. In this mode, the xrootd is free to
choose any port that is available if you did not specify a port number on
the command line or in the configuration file. The arbitrary port number
is then forwarded to the olbd. This allows the olbd to redirect clients to
the proper port even though itís not known ahead of time. This only works
if the olbd is not started with the -i option (the default) and the xrootd
is started with the -t option (required for data servers and supervisors).
This does not eliminate the need for starting the manager olbd and its
xrootd counterpart with well-known ports. We are working on automating
this process.

Does that mean I can use -a to run multiple data servers on a single node?

Yes. Again, you will need to manually manage the olbd administration path
and log files for each server. We are working on automating this process.

Can I use the -a option to prohibit clients to bypass the olbd?

Yes. This is actually recommended. Since arbitrary port numbers are
chosen, a client cannot directly connect to a data server without using
the manager xrootd. However, while significant programming effort is
required to capture port numbers at run-time; any "management by
obscurity" method can be defeated.