Hello,

> * Data loading :
>
>     - Could  you  define  how   data  will  be  generated  and
>       distributed on workers ?
For this test, we will make a copy of a smaller dataset (<100GB) on a 
shared filesystem mounted by the workers. We will then run a 
duplicator/partitioner on each worker which will synthesize the data 
portions that belong on that worker. The synthesizer is coded for a 
particular data distribution.

>
>     - Could  you begin  data  load testing  on  our 4  current
>       machines at CC ?
We can, if they have the same characteristics of machines in the target 
cluster. They must have the exact same installed packages (OS + 
libraries +environment), same disk configuration (free space mounted at 
the same mount points, similar disk characteristics), same memory size, 
and same kernel tweaks (max threads, max open files, etc.).

We have found that machines in a cluster run different configurations 
than general-purpose development machines (even when the OS is the 
same), and it often takes a couple days to shake out the problems, if 
the sysadmins are responsive (may be difficult across time zones). In 
the past, it has taken more than a week, but that was prior to having 
installation scripts.

It's probably easier for sysadmins to grant non-exclusive access to a 
few machines (having one or two exposes 80% of the issues) than to 
figure out how to make a dev machine look like a cluster machine.

>     - How far are Qserv installation and data loading
>       independent ?
I defer to Douglas. Qserv installation is strictly different than 
installing data. Loading data (creating a schema, partitioning data, 
loading it) is still a painful part of making data available via Qserv.


> * Production planning:
>
>     - What happens if a worker crash ? What actions are needed
>       ( error recovery ) ? How to exclude one worker node from
>       Qserv ?
If a worker crashes, we must login, debug it, and try to reproduce. 
Qserv has many bugs and the implementation is quite incomplete, and much 
of the value of this testing is to expose problems at scale. At minimum, 
we need the log files, the mysqld query log (should be enabled after 
data loading), and the stack trace of the xrootd process (available by 
saving the corefile).

Worker nodes register themselves, so if a worker node crashes, it will 
not respond to query dispatches. An administrator can lookup the data 
portions that reside on the worker node and mark them as bad (empty) on 
the master, allowing the master to complete, but return incorrect results.

>     - What exactly do you need to be monitored on workers ?
Log files, mostly. For debugging, we need free memory, free disk, cpu 
load, and mysqld processlist (output of SHOW PROCESSLIST, but mytop 
http://jeremy.zawodny.com/mysql/mytop/ would be really nice).

>     - We propose to  begin installation on 4, then  10, 50 and
>       finally 300 nodes. What  are the validation steps ? What
>       will be the queries used ?
I'm assuming that 4 = 1+3, 10= 1+9, and 300=1+299. Note that the master 
node may get heavily loaded in certain situations. The validation steps 
are simply to run queries: full-scan queries, point-lookup queries, and 
join queries. Each time, the workers need to be spot-checked to verify 
that they are responding as expected and the master log must be 
inspected, because errors may be unnoticeable in the query results.

> It would be good to have a wiki page for all these so that
> we could refer to it.
I think Douglas has started such a page, but we should check with him on 
tomorrow's call.

Qserv is quite complex and is often difficult to debug. The last test 
with 150 nodes would have been impossible without ssh access to the 
nodes for setup, validation, and debugging and responsive sysadmins to 
check for OS problems and faulty disks or other hardware (we had 2-3 
disk failures and 1 misbehaving machine in our 150 node test over 2 
weeks, and the more recent test had disk corruption and odd performance 
problems even at 16 nodes).

Note that some problems required new features to be quickly developed in 
order to continue testing. We will probably need such patches during 
this test.

We are really grateful for the opportunity to do this testing and hope 
to learn a lot about qserv, its future direction, and, more generally, 
the feasibility of the current plans for LSST catalog management.


Hope this helps,
-Daniel

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1