 |
 |
Hello, > * Data loading : > > - Could you define how data will be generated and > distributed on workers ? For this test, we will make a copy of a smaller dataset (<100GB) on a shared filesystem mounted by the workers. We will then run a duplicator/partitioner on each worker which will synthesize the data portions that belong on that worker. The synthesizer is coded for a particular data distribution. > > - Could you begin data load testing on our 4 current > machines at CC ? We can, if they have the same characteristics of machines in the target cluster. They must have the exact same installed packages (OS + libraries +environment), same disk configuration (free space mounted at the same mount points, similar disk characteristics), same memory size, and same kernel tweaks (max threads, max open files, etc.). We have found that machines in a cluster run different configurations than general-purpose development machines (even when the OS is the same), and it often takes a couple days to shake out the problems, if the sysadmins are responsive (may be difficult across time zones). In the past, it has taken more than a week, but that was prior to having installation scripts. It's probably easier for sysadmins to grant non-exclusive access to a few machines (having one or two exposes 80% of the issues) than to figure out how to make a dev machine look like a cluster machine. > - How far are Qserv installation and data loading > independent ? I defer to Douglas. Qserv installation is strictly different than installing data. Loading data (creating a schema, partitioning data, loading it) is still a painful part of making data available via Qserv. > * Production planning: > > - What happens if a worker crash ? What actions are needed > ( error recovery ) ? How to exclude one worker node from > Qserv ? If a worker crashes, we must login, debug it, and try to reproduce. Qserv has many bugs and the implementation is quite incomplete, and much of the value of this testing is to expose problems at scale. At minimum, we need the log files, the mysqld query log (should be enabled after data loading), and the stack trace of the xrootd process (available by saving the corefile). Worker nodes register themselves, so if a worker node crashes, it will not respond to query dispatches. An administrator can lookup the data portions that reside on the worker node and mark them as bad (empty) on the master, allowing the master to complete, but return incorrect results. > - What exactly do you need to be monitored on workers ? Log files, mostly. For debugging, we need free memory, free disk, cpu load, and mysqld processlist (output of SHOW PROCESSLIST, but mytop http://jeremy.zawodny.com/mysql/mytop/ would be really nice). > - We propose to begin installation on 4, then 10, 50 and > finally 300 nodes. What are the validation steps ? What > will be the queries used ? I'm assuming that 4 = 1+3, 10= 1+9, and 300=1+299. Note that the master node may get heavily loaded in certain situations. The validation steps are simply to run queries: full-scan queries, point-lookup queries, and join queries. Each time, the workers need to be spot-checked to verify that they are responding as expected and the master log must be inspected, because errors may be unnoticeable in the query results. > It would be good to have a wiki page for all these so that > we could refer to it. I think Douglas has started such a page, but we should check with him on tomorrow's call. Qserv is quite complex and is often difficult to debug. The last test with 150 nodes would have been impossible without ssh access to the nodes for setup, validation, and debugging and responsive sysadmins to check for OS problems and faulty disks or other hardware (we had 2-3 disk failures and 1 misbehaving machine in our 150 node test over 2 weeks, and the more recent test had disk corruption and odd performance problems even at 16 nodes). Note that some problems required new features to be quickly developed in order to continue testing. We will probably need such patches during this test. We are really grateful for the opportunity to do this testing and hope to learn a lot about qserv, its future direction, and, more generally, the feasibility of the current plans for LSST catalog management. Hope this helps, -Daniel ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1