Print

Print
H all,

Thanks to Andy’s fix, the data-loader is now working in multi-node even with my docker/openstack setup. Fabrice’s scripts also make the process of making this setup much quicker. We have some basic notes spread between me, John and Fabrice that we’ll consolidate into some documentation for this process. In principle, it took me only ~4 hours to update my repo with Andy’s fix, update the docker image with the stack and push it to the hub, then pull it to NCSA for the setup (while the upload of the ~4gb image was from my slower home internet connection).

At the moment, the stack/node setup itself seems to be fine and we are now at the point (running integration test) that qserv queries are failing (mysql queries run just fine). This is what I’m working on now and it is likely due to qserv/xrootd network configuration problems due to my test being on openstack.

-Vaikunth

On Nov 12, 2015, at 9:00 PM, Thukral, Vaikunth <[log in to unmask]> wrote:

Thanks Andy, having an easy multi-node test setup is exactly why we’re doing this =) . I’ll check your fix and update everyone tomorrow.

-Vaikunth


On Nov 12, 2015, at 7:09 PM, Salnikov, Andrei A. <[log in to unmask]> wrote:

Hi Vaikunth,

sorry for this problem, there was a bug in data loader which passed incorrect
object to one method. I have just piushed a fix to qserv master, try to use it.
 
Unfortunately we do not have an easy test for multi-node setup so this sort of
bugs does not show in our integration tests. Few people already suggested to
abandon mono-node setup and run integration tests in multi-node (on a single
host). I think this is a reasonable suggestion, we should try to see how easy
it is to implement that.

Cheers,
Andy
 
 
From: [log in to unmask] [mailto:[log in to unmask]] On Behalf Of Thukral, Vaikunth
Sent: Thursday, November 12, 2015 5:31 PM
To: qserv-l <[log in to unmask]>
Subject: [QSERV-L] CSS problem in multinode
 
Hi Andy, 
 
This is Vaikunth and Fabrice
We try to launch latest Qserv version on OpenStack/Docker but it seems there’s a missing python method getNodeParams in css module, this method seems to be called only in multinode mode. Here’s the log below.
 
We do not think the problem comes from Vaikunth code, and getNodeParams is in the code, but not in the python path (see call to python interpreter at bottom of the logs)
 
Could you help us please?
 
Thanks
 
 
 
qserv@vt-docker-1:~$ qserv-check-integration.py --case=01 --load
2015-11-13 01:25:27,713 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (mysql mode)
2015-11-13 01:25:27,713 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Object
2015-11-13 01:25:27,987 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Object
2015-11-13 01:25:27,988 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Filter
2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Filter
2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Science_Ccd_Exposure
2015-11-13 01:25:28,554 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Science_Ccd_Exposure
2015-11-13 01:25:28,555 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Logs
2015-11-13 01:25:28,832 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Logs
2015-11-13 01:25:28,833 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table LeapSeconds
2015-11-13 01:25:29,107 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table LeapSeconds
2015-11-13 01:25:29,108 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table SimRefObject
2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table SimRefObject
2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table sdqa_Metric
2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table sdqa_Metric
2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Source
2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Source
2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table RefSrcMatch
2015-11-13 01:25:30,283 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table RefSrcMatch
2015-11-13 01:25:30,284 - lsst.qserv.tests.benchmark - INFO - Launch 0001.1_fetchObjectByIdIN.sql against mysql
2015-11-13 01:25:30,290 - lsst.qserv.tests.benchmark - INFO - Launch 0001_fetchObjectById.sql against mysql
2015-11-13 01:25:30,297 - lsst.qserv.tests.benchmark - INFO - Launch 0002_fetchObjectByIdNoResult.sql against mysql
2015-11-13 01:25:30,303 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy.sql against mysql
2015-11-13 01:25:30,309 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_classicJOIN.sql against mysql
2015-11-13 01:25:30,315 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_withUSING.sql against mysql
2015-11-13 01:25:30,322 - lsst.qserv.tests.benchmark - INFO - Launch 0005_nonReplicatedTable.sql against mysql
2015-11-13 01:25:30,327 - lsst.qserv.tests.benchmark - INFO - Launch 0007_fetchSourceByObjId.sql against mysql
2015-11-13 01:25:30,333 - lsst.qserv.tests.benchmark - INFO - Launch 0008.1_fetchSourceByObjIdIN_noRes.sql against mysql
2015-11-13 01:25:30,339 - lsst.qserv.tests.benchmark - INFO - Launch 0008.2_fetchSourceByObjIdIN_withRes.sql against mysql
2015-11-13 01:25:30,344 - lsst.qserv.tests.benchmark - INFO - Launch 0008.3_fetchSourceByObjIdIN.sql against mysql
2015-11-13 01:25:30,350 - lsst.qserv.tests.benchmark - INFO - Launch 0008.4_fetchSourceByObjIdBETWEEN.sql against mysql
2015-11-13 01:25:30,356 - lsst.qserv.tests.benchmark - INFO - Launch 0008_fetchSourceByObjIdIN_withRes.sql against mysql
2015-11-13 01:25:30,362 - lsst.qserv.tests.benchmark - INFO - Launch 0012.1_raftAndCcd.sql against mysql
2015-11-13 01:25:30,369 - lsst.qserv.tests.benchmark - INFO - Launch 0012.2_raftAndCcd.sql against mysql
2015-11-13 01:25:30,376 - lsst.qserv.tests.benchmark - INFO - Launch 0012_raftAndCcd.sql against mysql
2015-11-13 01:25:30,383 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor.sql against mysql
2015-11-13 01:25:30,389 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor_noalias.sql against mysql
2015-11-13 01:25:30,394 - lsst.qserv.tests.benchmark - INFO - Launch 1003_coneMagColorEmptyRes.sql against mysql
2015-11-13 01:25:30,400 - lsst.qserv.tests.benchmark - INFO - Launch 1004.1_varObjects.sql against mysql
2015-11-13 01:25:30,406 - lsst.qserv.tests.benchmark - INFO - Launch 1004_varObjects.sql against mysql
2015-11-13 01:25:30,411 - lsst.qserv.tests.benchmark - INFO - Launch 1011_objectsForExposure.sql against mysql
2015-11-13 01:25:30,423 - lsst.qserv.tests.benchmark - INFO - Launch 1030_timeSeries.sql against mysql
2015-11-13 01:25:30,431 - lsst.qserv.tests.benchmark - INFO - Launch 1051_nn.sql against mysql
2015-11-13 01:25:30,437 - lsst.qserv.tests.benchmark - INFO - Launch 1060_selectPoly.sql against mysql
2015-11-13 01:25:30,456 - lsst.qserv.tests.benchmark - INFO - Launch 3006.1_selectAs.sql against mysql
2015-11-13 01:25:30,461 - lsst.qserv.tests.benchmark - INFO - Launch 3006_selectAs.sql against mysql
2015-11-13 01:25:30,467 - lsst.qserv.tests.benchmark - INFO - Launch 9000_caseinsensitiveUDF.sql against mysql
2015-11-13 01:25:30,472 - lsst.qserv.tests.benchmark - INFO - Test case #01: 28 queries launched on a total of 68
2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker1
2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker2
2015-11-13 01:25:30,476 - lsst.qserv.tests.qservDbLoader - INFO - Drop and create MySQL database for Qserv: qservTest_case01_qserv
2015-11-13 01:25:30,542 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database for Qserv
2015-11-13 01:25:30,543 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database: qservTest_case01_qserv
2015-11-13 01:25:30,544 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (qserv mode)
2015-11-13 01:25:30,544 - lsst.qserv.tests.qservDbLoader - INFO - Partition data, create and load table Object
2015-11-13 01:25:30,756 - root - CRITICAL - Exception occured: 'module' object has no attribute 'getNodeParams'
Traceback (most recent call last):
  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 274, in <module>
    loader = Loader()
  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 230, in __init__
    workerWmgrMap[worker] = self._wmgrConnect(worker, css)
  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 265, in _wmgrConnect
    nodeAdmin = NodeAdmin(name=nodeName, css=css, wmgrSecretFile=self.args.secret)
  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/lib/python/lsst/qserv/admin/nodeAdmin.py", line 94, in __init__
    params = css.getNodeParams(name)
AttributeError: 'module' object has no attribute 'getNodeParams'
2015-11-13 01:25:30,768 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : qserv-data-loader.py -v --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/common.cfg --host=vt-docker-1 --port=5012 --secret=/qserv/run/etc/wmgr.secret --delete-tables --chunks-dir=/qserv/run/tmp/qserv_data_loader/Object --css-remove --worker worker1 --worker worker2 --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.cfg --empty-chunks=/qserv/run/var/lib/qserv/empty_qservTest_case01_qserv.txt qservTest_case01_qserv Object /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.schema /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.tsv.gz 
 
 
qserv@vt-docker-1:~$ python                                     
Python 2.7.9 |Anaconda 2.2.0 (64-bit)| (default, Mar  9 2015, 16:20:48) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
>>> from lsst.qserv import css
>>> dir(css)
['AuthError', 'BadAllocError', 'ChunkMap', 'ChunkMap_swigregister', 'ConfigError', 'ConnError', 'CssAccess', 'CssAccess_createFromConfig', 'CssAccess_createFromData', 'CssAccess_createFromStream', 'CssAccess_cssVersion', 'CssAccess_swigregister', 'CssError', 'EmptyChunks', 'EmptyChunks_swigregister', 'KEY_STATUS_CREATE_PFX', 'KEY_STATUS_DROP_PFX', 'KEY_STATUS_FAILED_PFX', 'KEY_STATUS_IGNORE', 'KEY_STATUS_READY', 'KeyExistsError', 'KeyValueError', 'KvInterface', 'KvInterface_swigregister', 'MatchTableParams', 'MatchTableParams_swigregister', 'NODE_STATE_ACTIVE', 'NODE_STATE_INACTIVE', 'NoSuchDb', 'NoSuchKey', 'NoSuchNode', 'NoSuchTable', 'NodeExists', 'NodeInUse', 'NodeParamMap', 'NodeParamMap_swigregister', 'NodeParams', 'NodeParams_swigregister', 'PartTableParams', 'PartTableParams_swigregister', 'ReadonlyCss', 'SHARED_PTR_DISOWN', 'SqlErrorObject', 'SqlErrorObject_swigregister', 'StringMap', 'StringMap_swigregister', 'StringVector', 'StringVector_swigregister', 'StripingParams', 'StripingParams_swigregister', 'SwigPyIterator', 'SwigPyIterator_swigregister', 'TableExists', 'TableParams', 'TableParams_swigregister', 'VERSION', 'VERSION_KEY', 'VERSION_STR', 'VersionMismatchError', 'VersionMissingError', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'cssLib', 'cvar']
 

Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1



Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1



Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1