Hi Vaikunth,
sorry for this problem, there was a bug in data loader which passed incorrect
object to one method. I have just piushed a fix to qserv master, try to use it.
Unfortunately we do not have an easy test for multi-node setup so this sort of
bugs does not show in our integration tests. Few people already suggested to
abandon mono-node setup and run integration tests in multi-node (on a single
host). I think this is a reasonable suggestion, we should try to see how easy
it is to implement that.
Cheers,
Andy
From: [log in to unmask] [mailto:[log in to unmask]] On Behalf Of Thukral, Vaikunth
Sent: Thursday, November 12, 2015 5:31 PM
To: qserv-l <[log in to unmask]>
Subject: [QSERV-L] CSS problem in multinode
Hi Andy,
This is Vaikunth and Fabrice
We try to launch latest Qserv version on OpenStack/Docker but it seems there’s a missing python method getNodeParams in css module, this method seems to be called only in multinode mode. Here’s the log below.
We do not think the problem comes from Vaikunth code, and getNodeParams is in the code, but not in the python path (see call to python interpreter at bottom of the logs)
Could you help us please?
Thanks
qserv@vt-docker-1:~$ qserv-check-integration.py --case=01 --load
2015-11-13 01:25:27,713 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (mysql mode)
2015-11-13 01:25:27,713 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Object
2015-11-13 01:25:27,987 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Object
2015-11-13 01:25:27,988 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Filter
2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Filter
2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Science_Ccd_Exposure
2015-11-13 01:25:28,554 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Science_Ccd_Exposure
2015-11-13 01:25:28,555 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Logs
2015-11-13 01:25:28,832 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Logs
2015-11-13 01:25:28,833 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table LeapSeconds
2015-11-13 01:25:29,107 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table LeapSeconds
2015-11-13 01:25:29,108 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table SimRefObject
2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table SimRefObject
2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table sdqa_Metric
2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table sdqa_Metric
2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Source
2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Source
2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table RefSrcMatch
2015-11-13 01:25:30,283 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table RefSrcMatch
2015-11-13 01:25:30,284 - lsst.qserv.tests.benchmark - INFO - Launch 0001.1_fetchObjectByIdIN.sql against mysql
2015-11-13 01:25:30,290 - lsst.qserv.tests.benchmark - INFO - Launch 0001_fetchObjectById.sql against mysql
2015-11-13 01:25:30,297 - lsst.qserv.tests.benchmark - INFO - Launch 0002_fetchObjectByIdNoResult.sql against mysql
2015-11-13 01:25:30,303 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy.sql against mysql
2015-11-13 01:25:30,309 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_classicJOIN.sql against mysql
2015-11-13 01:25:30,315 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_withUSING.sql against mysql
2015-11-13 01:25:30,322 - lsst.qserv.tests.benchmark - INFO - Launch 0005_nonReplicatedTable.sql against mysql
2015-11-13 01:25:30,327 - lsst.qserv.tests.benchmark - INFO - Launch 0007_fetchSourceByObjId.sql against mysql
2015-11-13 01:25:30,333 - lsst.qserv.tests.benchmark - INFO - Launch 0008.1_fetchSourceByObjIdIN_noRes.sql against mysql
2015-11-13 01:25:30,339 - lsst.qserv.tests.benchmark - INFO - Launch 0008.2_fetchSourceByObjIdIN_withRes.sql against mysql
2015-11-13 01:25:30,344 - lsst.qserv.tests.benchmark - INFO - Launch 0008.3_fetchSourceByObjIdIN.sql against mysql
2015-11-13 01:25:30,350 - lsst.qserv.tests.benchmark - INFO - Launch 0008.4_fetchSourceByObjIdBETWEEN.sql against mysql
2015-11-13 01:25:30,356 - lsst.qserv.tests.benchmark - INFO - Launch 0008_fetchSourceByObjIdIN_withRes.sql against mysql
2015-11-13 01:25:30,362 - lsst.qserv.tests.benchmark - INFO - Launch 0012.1_raftAndCcd.sql against mysql
2015-11-13 01:25:30,369 - lsst.qserv.tests.benchmark - INFO - Launch 0012.2_raftAndCcd.sql against mysql
2015-11-13 01:25:30,376 - lsst.qserv.tests.benchmark - INFO - Launch 0012_raftAndCcd.sql against mysql
2015-11-13 01:25:30,383 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor.sql against mysql
2015-11-13 01:25:30,389 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor_noalias.sql against mysql
2015-11-13 01:25:30,394 - lsst.qserv.tests.benchmark - INFO - Launch 1003_coneMagColorEmptyRes.sql against mysql
2015-11-13 01:25:30,400 - lsst.qserv.tests.benchmark - INFO - Launch 1004.1_varObjects.sql against mysql
2015-11-13 01:25:30,406 - lsst.qserv.tests.benchmark - INFO - Launch 1004_varObjects.sql against mysql
2015-11-13 01:25:30,411 - lsst.qserv.tests.benchmark - INFO - Launch 1011_objectsForExposure.sql against mysql
2015-11-13 01:25:30,423 - lsst.qserv.tests.benchmark - INFO - Launch 1030_timeSeries.sql against mysql
2015-11-13 01:25:30,431 - lsst.qserv.tests.benchmark - INFO - Launch 1051_nn.sql against mysql
2015-11-13 01:25:30,437 - lsst.qserv.tests.benchmark - INFO - Launch 1060_selectPoly.sql against mysql
2015-11-13 01:25:30,456 - lsst.qserv.tests.benchmark - INFO - Launch 3006.1_selectAs.sql against mysql
2015-11-13 01:25:30,461 - lsst.qserv.tests.benchmark - INFO - Launch 3006_selectAs.sql against mysql
2015-11-13 01:25:30,467 - lsst.qserv.tests.benchmark - INFO - Launch 9000_caseinsensitiveUDF.sql against mysql
2015-11-13 01:25:30,472 - lsst.qserv.tests.benchmark - INFO - Test case #01: 28 queries launched on a total of 68
2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker1
2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker2
2015-11-13 01:25:30,476 - lsst.qserv.tests.qservDbLoader - INFO - Drop and create MySQL database for Qserv: qservTest_case01_qserv
2015-11-13 01:25:30,542 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database for Qserv
2015-11-13 01:25:30,543 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database: qservTest_case01_qserv
2015-11-13 01:25:30,544 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (qserv mode)
2015-11-13 01:25:30,544 - lsst.qserv.tests.qservDbLoader - INFO - Partition data, create and load table Object
2015-11-13 01:25:30,756 - root - CRITICAL - Exception occured: 'module' object has no attribute 'getNodeParams'
Traceback (most recent call last):
File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 274, in <module>
loader = Loader()
File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 230, in __init__
workerWmgrMap[worker] = self._wmgrConnect(worker, css)
File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 265, in _wmgrConnect
nodeAdmin = NodeAdmin(name=nodeName, css=css, wmgrSecretFile=self.args.secret)
File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/lib/python/lsst/qserv/admin/nodeAdmin.py", line 94, in __init__
params = css.getNodeParams(name)
AttributeError: 'module' object has no attribute 'getNodeParams'
2015-11-13 01:25:30,768 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : qserv-data-loader.py -v --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/common.cfg --host=vt-docker-1 --port=5012 --secret=/qserv/run/etc/wmgr.secret --delete-tables --chunks-dir=/qserv/run/tmp/qserv_data_loader/Object --css-remove --worker worker1 --worker worker2 --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.cfg --empty-chunks=/qserv/run/var/lib/qserv/empty_qservTest_case01_qserv.txt qservTest_case01_qserv Object /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.schema /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.tsv.gz
qserv@vt-docker-1:~$ python
Python 2.7.9 |Anaconda 2.2.0 (64-bit)| (default, Mar 9 2015, 16:20:48)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> from lsst.qserv import css
>>> dir(css)
['AuthError', 'BadAllocError', 'ChunkMap', 'ChunkMap_swigregister', 'ConfigError', 'ConnError', 'CssAccess', 'CssAccess_createFromConfig', 'CssAccess_createFromData', 'CssAccess_createFromStream', 'CssAccess_cssVersion', 'CssAccess_swigregister', 'CssError', 'EmptyChunks', 'EmptyChunks_swigregister', 'KEY_STATUS_CREATE_PFX', 'KEY_STATUS_DROP_PFX', 'KEY_STATUS_FAILED_PFX', 'KEY_STATUS_IGNORE', 'KEY_STATUS_READY', 'KeyExistsError', 'KeyValueError', 'KvInterface', 'KvInterface_swigregister', 'MatchTableParams', 'MatchTableParams_swigregister', 'NODE_STATE_ACTIVE', 'NODE_STATE_INACTIVE', 'NoSuchDb', 'NoSuchKey', 'NoSuchNode', 'NoSuchTable', 'NodeExists', 'NodeInUse', 'NodeParamMap', 'NodeParamMap_swigregister', 'NodeParams', 'NodeParams_swigregister', 'PartTableParams', 'PartTableParams_swigregister', 'ReadonlyCss', 'SHARED_PTR_DISOWN', 'SqlErrorObject', 'SqlErrorObject_swigregister', 'StringMap', 'StringMap_swigregister', 'StringVector', 'StringVector_swigregister', 'StripingParams', 'StripingParams_swigregister', 'SwigPyIterator', 'SwigPyIterator_swigregister', 'TableExists', 'TableParams', 'TableParams_swigregister', 'VERSION', 'VERSION_KEY', 'VERSION_STR', 'VersionMismatchError', 'VersionMissingError', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'cssLib', 'cvar']
________________________________
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
