 |
 |
Hello, In the minutes of your last meeting, we've seen that you are trying to load W13 data in Qserv. This interests us a lot because we may plan to use this dataset on CCIN2P3 250 nodes cluster. As asked by Jacek, we've worked on tests/runTests.py script and it seems to works quite well now (even if some improvments are still needed). That's why Emmanuel Gangler subtly suggests us to give you some feedback about this testing procedure, because in fact it contains an algorithm which partition/load/generate meta for a small dataset. In tests/runTests.py, the function loadPartitionedData() has been rewrited w.r.t. Jacek comments, and it seems it could be interesting to generalize it for various LSST data. It relies on partition.py (a bit patched for tsv support) and load.py (patched with ugly debug messages) I will be on holidays next week, but Emmanuel Medernach may work on it, and study how to partition, load, and generate meta for PT1.1 Source and Object data with this function. I hope and think that loadPartitionedData() algorithm should't be to much impacted by this generalization, but only the configuration parameters of the input data (hard-coded at the beginning of the function, but which could be easily readable from a config file). If this algorithm could be used for data loading, some interesting job would also remains in order to distribute loading, and why not, meta-generation. Hope this helps. Cheers, Fabrice P.S. : You can try it by launching tests/runTests.py (assuming you've got a qserv.conf in your #/.lsst dir, as explained in README.txt). (In addtion this ticket has been merged with a templating system update, and a minor bugfix in qserv-stop in order to stop also mysqld, cmsd and xrootd). ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1