 |
 |
We will discuss this at the meeting, but here are some answers in writing to use later also: On 06/06/2013 08:13 AM, Emmanuel Medernach wrote: > Hello, > > Rachid sends us a list of questions below. > > * Data loading : > > - Could you define how data will be generated and > distributed on workers ? For this test the data will be generated on each node. It will use data for input, what we call the "pt12" data, which is already there at in2p3 in an nfs volume. This input data will be used to create separated sets on each node. > > - Could you begin data load testing on our 4 current > machines at CC ? Yes, we can, but we also need to test machines that are exactly the same as the test. So we can test the data size on the local storage, the amount of cpu used and time needed, and networking to the machines. > > - How far are Qserv installation and data loading > independent ? Well, the qserv install will have to be complete before the data creation and loading can start. But the qserv install should only take a couple hours really, the data creation will take days or so. > > * Production planning: > > - What happens if a worker crash ? What actions are needed > ( error recovery ) ? How to exclude one worker node from > Qserv ? If a worker goes down, then we will take that as data loss, and not include that in the test. For completeness tests then we can adjust the number of "chunks" in use, and restart the master and continue the tests. > > - What exactly do you need to be monitored on workers ? Depends on what you want to setup for monitoring. We will need to be able to check on the processes is use on each node as needed, and perhaps check logs on a node as needed. It would be nice to be able to monitor the cpu use and local storage i/o rates if possible, if you have that. > > - We propose to begin installation on 4, then 10, 50 and > finally 300 nodes. What are the validation steps ? What > will be the queries used ? For the install Emmanuel and Fabrice have the install tests, then we can start the data creation. There we will check that that those processes finish without any errors. That data needs to be loaded into the mysql layer, and that needs to finish without errors. Then we can start up the qserv master and check on the access to the worker nodes. This always takes some debugging the data access. But then querries can get done to check on the data access. > > > It would be good to have a wiki page for all these so that > we could refer to it. Yes, sure, lets start one. > > We could discuss about this during our tomorrow meeting. > > -- > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1