Print

Print

Hi Vaikunth,

sorry for this problem, there was a bug in data loader which passed incorrect

object to one method. I have just piushed a fix to qserv master, try to use it.

 

Unfortunately we do not have an easy test for multi-node setup so this sort of

bugs does not show in our integration tests. Few people already suggested to

abandon mono-node setup and run integration tests in multi-node (on a single

host). I think this is a reasonable suggestion, we should try to see how easy

it is to implement that.

Cheers,
Andy
 

 

From: [log in to unmask] [mailto:[log in to unmask]] On Behalf Of Thukral, Vaikunth
Sent: Thursday, November 12, 2015 5:31 PM
To: qserv-l <[log in to unmask]>
Subject: [QSERV-L] CSS problem in multinode

 

Hi Andy,

 

This is Vaikunth and Fabrice

We try to launch latest Qserv version on OpenStack/Docker but it seems there’s a missing python method getNodeParams in css module, this method seems to be called only in multinode mode. Here’s the log below.

 

We do not think the problem comes from Vaikunth code, and getNodeParams is in the code, but not in the python path (see call to python interpreter at bottom of the logs)

 

Could you help us please?

 

Thanks

 

 

 

qserv@vt-docker-1:~$ qserv-check-integration.py --case=01 --load

2015-11-13 01:25:27,713 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (mysql mode)

2015-11-13 01:25:27,713 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Object

2015-11-13 01:25:27,987 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Object

2015-11-13 01:25:27,988 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Filter

2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Filter

2015-11-13 01:25:28,243 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Science_Ccd_Exposure

2015-11-13 01:25:28,554 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Science_Ccd_Exposure

2015-11-13 01:25:28,555 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Logs

2015-11-13 01:25:28,832 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Logs

2015-11-13 01:25:28,833 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table LeapSeconds

2015-11-13 01:25:29,107 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table LeapSeconds

2015-11-13 01:25:29,108 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table SimRefObject

2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table SimRefObject

2015-11-13 01:25:29,394 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table sdqa_Metric

2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table sdqa_Metric

2015-11-13 01:25:29,656 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table Source

2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table Source

2015-11-13 01:25:30,001 - lsst.qserv.tests.mysqlDbLoader - INFO - Create, load table RefSrcMatch

2015-11-13 01:25:30,283 - lsst.qserv.tests.mysqlDbLoader - INFO - Partitioned data loaded for table RefSrcMatch

2015-11-13 01:25:30,284 - lsst.qserv.tests.benchmark - INFO - Launch 0001.1_fetchObjectByIdIN.sql against mysql

2015-11-13 01:25:30,290 - lsst.qserv.tests.benchmark - INFO - Launch 0001_fetchObjectById.sql against mysql

2015-11-13 01:25:30,297 - lsst.qserv.tests.benchmark - INFO - Launch 0002_fetchObjectByIdNoResult.sql against mysql

2015-11-13 01:25:30,303 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy.sql against mysql

2015-11-13 01:25:30,309 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_classicJOIN.sql against mysql

2015-11-13 01:25:30,315 - lsst.qserv.tests.benchmark - INFO - Launch 0003_selectMetadataForOneGalaxy_withUSING.sql against mysql

2015-11-13 01:25:30,322 - lsst.qserv.tests.benchmark - INFO - Launch 0005_nonReplicatedTable.sql against mysql

2015-11-13 01:25:30,327 - lsst.qserv.tests.benchmark - INFO - Launch 0007_fetchSourceByObjId.sql against mysql

2015-11-13 01:25:30,333 - lsst.qserv.tests.benchmark - INFO - Launch 0008.1_fetchSourceByObjIdIN_noRes.sql against mysql

2015-11-13 01:25:30,339 - lsst.qserv.tests.benchmark - INFO - Launch 0008.2_fetchSourceByObjIdIN_withRes.sql against mysql

2015-11-13 01:25:30,344 - lsst.qserv.tests.benchmark - INFO - Launch 0008.3_fetchSourceByObjIdIN.sql against mysql

2015-11-13 01:25:30,350 - lsst.qserv.tests.benchmark - INFO - Launch 0008.4_fetchSourceByObjIdBETWEEN.sql against mysql

2015-11-13 01:25:30,356 - lsst.qserv.tests.benchmark - INFO - Launch 0008_fetchSourceByObjIdIN_withRes.sql against mysql

2015-11-13 01:25:30,362 - lsst.qserv.tests.benchmark - INFO - Launch 0012.1_raftAndCcd.sql against mysql

2015-11-13 01:25:30,369 - lsst.qserv.tests.benchmark - INFO - Launch 0012.2_raftAndCcd.sql against mysql

2015-11-13 01:25:30,376 - lsst.qserv.tests.benchmark - INFO - Launch 0012_raftAndCcd.sql against mysql

2015-11-13 01:25:30,383 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor.sql against mysql

2015-11-13 01:25:30,389 - lsst.qserv.tests.benchmark - INFO - Launch 1002_coneMagColor_noalias.sql against mysql

2015-11-13 01:25:30,394 - lsst.qserv.tests.benchmark - INFO - Launch 1003_coneMagColorEmptyRes.sql against mysql

2015-11-13 01:25:30,400 - lsst.qserv.tests.benchmark - INFO - Launch 1004.1_varObjects.sql against mysql

2015-11-13 01:25:30,406 - lsst.qserv.tests.benchmark - INFO - Launch 1004_varObjects.sql against mysql

2015-11-13 01:25:30,411 - lsst.qserv.tests.benchmark - INFO - Launch 1011_objectsForExposure.sql against mysql

2015-11-13 01:25:30,423 - lsst.qserv.tests.benchmark - INFO - Launch 1030_timeSeries.sql against mysql

2015-11-13 01:25:30,431 - lsst.qserv.tests.benchmark - INFO - Launch 1051_nn.sql against mysql

2015-11-13 01:25:30,437 - lsst.qserv.tests.benchmark - INFO - Launch 1060_selectPoly.sql against mysql

2015-11-13 01:25:30,456 - lsst.qserv.tests.benchmark - INFO - Launch 3006.1_selectAs.sql against mysql

2015-11-13 01:25:30,461 - lsst.qserv.tests.benchmark - INFO - Launch 3006_selectAs.sql against mysql

2015-11-13 01:25:30,467 - lsst.qserv.tests.benchmark - INFO - Launch 9000_caseinsensitiveUDF.sql against mysql

2015-11-13 01:25:30,472 - lsst.qserv.tests.benchmark - INFO - Test case #01: 28 queries launched on a total of 68

2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker1

2015-11-13 01:25:30,475 - lsst.qserv.tests.qservDbLoader - INFO - Running on node: worker2

2015-11-13 01:25:30,476 - lsst.qserv.tests.qservDbLoader - INFO - Drop and create MySQL database for Qserv: qservTest_case01_qserv

2015-11-13 01:25:30,542 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database for Qserv

2015-11-13 01:25:30,543 - lsst.qserv.tests.qservDbLoader - INFO - Drop CSS database: qservTest_case01_qserv

2015-11-13 01:25:30,544 - lsst.qserv.tests.benchmark - INFO - Loading data from /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data (qserv mode)

2015-11-13 01:25:30,544 - lsst.qserv.tests.qservDbLoader - INFO - Partition data, create and load table Object

2015-11-13 01:25:30,756 - root - CRITICAL - Exception occured: 'module' object has no attribute 'getNodeParams'

Traceback (most recent call last):

  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 274, in <module>

    loader = Loader()

  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 230, in __init__

    workerWmgrMap[worker] = self._wmgrConnect(worker, css)

  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/bin/qserv-data-loader.py", line 265, in _wmgrConnect

    nodeAdmin = NodeAdmin(name=nodeName, css=css, wmgrSecretFile=self.args.secret)

  File "/qserv/stack/Linux64/qserv/u.vaikunth.test_sqlconn-gcef6291637/lib/python/lsst/qserv/admin/nodeAdmin.py", line 94, in __init__

    params = css.getNodeParams(name)

AttributeError: 'module' object has no attribute 'getNodeParams'

2015-11-13 01:25:30,768 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : qserv-data-loader.py -v --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/common.cfg --host=vt-docker-1 --port=5012 --secret=/qserv/run/etc/wmgr.secret --delete-tables --chunks-dir=/qserv/run/tmp/qserv_data_loader/Object --css-remove --worker worker1 --worker worker2 --config=/qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.cfg --empty-chunks=/qserv/run/var/lib/qserv/empty_qservTest_case01_qserv.txt qservTest_case01_qserv Object /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.schema /qserv/stack/Linux64/qserv_testdata/2015_08.0-4-gc5b93d0+14/datasets/case01/data/Object.tsv.gz 

 

 

qserv@vt-docker-1:~$ python                                     

Python 2.7.9 |Anaconda 2.2.0 (64-bit)| (default, Mar  9 2015, 16:20:48) 

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

Anaconda is brought to you by Continuum Analytics.

Please check out: http://continuum.io/thanks and https://binstar.org

>>> from lsst.qserv import css

>>> dir(css)

['AuthError', 'BadAllocError', 'ChunkMap', 'ChunkMap_swigregister', 'ConfigError', 'ConnError', 'CssAccess', 'CssAccess_createFromConfig', 'CssAccess_createFromData', 'CssAccess_createFromStream', 'CssAccess_cssVersion', 'CssAccess_swigregister', 'CssError', 'EmptyChunks', 'EmptyChunks_swigregister', 'KEY_STATUS_CREATE_PFX', 'KEY_STATUS_DROP_PFX', 'KEY_STATUS_FAILED_PFX', 'KEY_STATUS_IGNORE', 'KEY_STATUS_READY', 'KeyExistsError', 'KeyValueError', 'KvInterface', 'KvInterface_swigregister', 'MatchTableParams', 'MatchTableParams_swigregister', 'NODE_STATE_ACTIVE', 'NODE_STATE_INACTIVE', 'NoSuchDb', 'NoSuchKey', 'NoSuchNode', 'NoSuchTable', 'NodeExists', 'NodeInUse', 'NodeParamMap', 'NodeParamMap_swigregister', 'NodeParams', 'NodeParams_swigregister', 'PartTableParams', 'PartTableParams_swigregister', 'ReadonlyCss', 'SHARED_PTR_DISOWN', 'SqlErrorObject', 'SqlErrorObject_swigregister', 'StringMap', 'StringMap_swigregister', 'StringVector', 'StringVector_swigregister', 'StripingParams', 'StripingParams_swigregister', 'SwigPyIterator', 'SwigPyIterator_swigregister', 'TableExists', 'TableParams', 'TableParams_swigregister', 'VERSION', 'VERSION_KEY', 'VERSION_STR', 'VersionMismatchError', 'VersionMissingError', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'cssLib', 'cvar']

 

Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1


Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1