There is no easy way to trnser data from zookeeper CSS into mysql,
but I can try to make a simple script to do that. Do you still have
zookeeper running? I need a dump of the old CSS, simplest is to
run qserv-admin from old release (with zookeeper) and say
"DUMP EVERYHTING;". Send the output to me so I can play with that.
Cheers,
Andy
From: Gates, John H
Sent: Tuesday, November 10, 2015 1:44 PM
To: Salnikov, Andrei A.; Fabrice Jammes; qserv-l
Subject: Re: [QSERV-L] Upgrade of IN2P3 css metadata
Hi Andy,
We had to ran this command after yours:
mysql -S /qserv/run/var/lib/mysql/mysql.sock -u root -pchangeme -e "GRANT ALL ON qservCssData.* TO 'qsmaster'@'localhost';"
Configuration files where correct because they have been generated with latest configuration procedure (but not the data which is in2p3 35TB dataset).
But now we have this error message:
ERROR 4110 (Proxy): Query processing error: "NoSuchTable:Table 'LSST.Object' does not exist."
And this error log:
==> /qserv/run/var/log/mysql-proxy-lua.log <==
*******************
Intercepted: select count(*) from Object
proxy.connection.server.thread_id: 7
Passing query: select count(*) from Object
Passing hints: "client_dst_name" "172.17.0.12:4040" "server_thread_id" "7" "db" "LSST"
Sendresult -4110
ERROR errNo: 4110: errMsg: Query processing error: "NoSuchTable:Table 'LSST.Object' does not exist."
==> /qserv/run/var/log/qserv-czar.log <==
2015-11-10T21:33:23.659Z [0x7f90b72ff700] ERROR ccontrol.UserQueryFactory (core/modules/ccontrol/UserQueryFactory.cc:101) - Invalid query: NoSuchTable:Table 'LSST.Object' does not exist.
2015-11-10T21:33:23.659Z [0x7f90b72ff700] INFO root (core/modules/ccontrol/UserQuery.cc:275) - Discarded UserQuery(6)
2015-11-10T21:33:23.662Z [0x7f90b72ff700] ERROR root (appInterface.py:138) - Returning error to proxy: ('"NoSuchTable:Table \'LSST.Object\' does not exist."', False, False, False)
The problem is that we are trying to use in2p3 35TB dataset, and it doesn't have CSS metadata at new format, but zookeeper one. Do you know how we can load this metadata without impacting data please?
This pecore/modules/css/CssAccess.cc:447: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:482: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:501: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:531: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:577: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:712: throw NoSuchTable(dbName, tableName);
core/modules/css/CssAccess.cc:902: if (not _kvI->exists(tableKey)) throw NoSuchTable(dbName, tableName);ace of code seems to be looking for CSS data which doesn't exist:
Thanks :-)
On 11/10/15 11:58, Salnikov, Andrei A. wrote:
Hi Fabrice,
it looks likely due to missing database for CSS data.
Did you swich to the latest qserv release already? Did
you run qserv-configure when you did that?
If you do not want to run qserv-configure with a new release
you need to do some manual work for updating your setup:
- create database and tables for CSS:
$ mysql -S /path/to/socket/file -uroot -p*** < admin/templates/configuration/tmp/configure/sql/CssData.sql
- update czar config file with CSS options, section [css]
should look like:
[css]
technology=mysql
hostname =
port =
username = qsmaster
password =
database = qservCssData
socket = /czar_instance_run_dir/var/lib/mysql/mysql.sock
- restart czar
Cheers,
Andy
From: [log in to unmask]<mailto:[log in to unmask]> [mailto:[log in to unmask]] On Behalf Of Fabrice Jammes
Sent: Tuesday, November 10, 2015 9:42 AM
To: qserv-l
Subject: [QSERV-L] Upgrade of IN2P3 css metadata
Hello,
CSS metadata is still stored in zookeeper on cc-in2p3 cluster, so any SQL query on a partitioned table crashes Qserv.
Does somebody knows how to upgrade css metadata to new format?
If it is proven that the crash is due to wrong metadata format, I'll open a bug, this shouldn't crash Qserv but print explicit log message instead, isn't i?
Error report is available below.
Thanks,
-------- Forwarded Message --------
Subject:
About docker install
Date:
Mon, 09 Nov 2015 16:58:23 -0800
From:
Fabrice Jammes <[log in to unmask]><mailto:[log in to unmask]>
To:
John Gates <[log in to unmask]><mailto:[log in to unmask]>
Hi John,
It seems our docker configuration is working a bit:
fjammes@ccosvms0070:~$ time mysql --host ccqserv125 --port 4040 --user
qsmaster LSST -e "show tables;";
+------------------------------------+
| Tables_in_LSST |
+------------------------------------+
| ForcedSource |
| ForcedSourceFullOverlap_1234567890 |
| ForcedSource_1234567890 |
| Object |
| ObjectFullOverlap_1234567890 |
| Object_1234567890 |
| Source |
| SourceFullOverlap_1234567890 |
| Source_1234567890 |
+------------------------------------+
real 0m0.027s
user 0m0.010s
sys 0m0.013s
Nevertheless next query fails and crash qserv czar:
fjammes@ccosvms0070:~$ time mysql --host ccqserv125 --port 4040 --user
qsmaster LSST -e "select count(*) from Object;";
ERROR 4005 (Proxy) at line 1: Unable to run lua xmlrpc client, message:
...inux64/lua/5.1.4.lsst1/share/lua/5.1/xmlrpc/http.lua:41: closed
real 0m0.105s
user 0m0.004s
sys 0m0.009s
qserv-czar.log doesn't print any error message:
qserv@ccqserv125:/qserv$ tail /qserv/run/var/log/qserv-czar.log
user = qsmaster
port = 0
2015-11-10T00:53:52.413Z [0x7f98d86af700] DEBUG root
(appInterface.py:70) - _idCounter 27996832412
2015-11-10T00:53:52.413Z [0x7f98d86af700] INFO root (server.py:137) -
endpoints: [<bound method AppInterface.cancelEverything of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.check of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.joinQuery of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.killQuery of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.killQueryUgly of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.resultTableString of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.results of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method AppInterface.submitQuery of
<lsst.qserv.czar.appInterface.AppInterface instance at 0x7f98cba83878>>,
<bound method Czar.echo of <lsst.qserv.czar.server.Czar instance at
0x7f98cba83830>>, <bound method Czar.stop of
<lsst.qserv.czar.server.Czar instance at 0x7f98cba83830>>, <bound method
Czar.reset of <lsst.qserv.czar.server.Czar instance at 0x7f98cba83830>>]
2015-11-10T00:53:52.420Z [0x7f98d86af700] INFO root (server.py:128) -
Starting Qserv interface on port: 7080
2015-11-10T00:56:40.602Z [0x7f98d86af700] DEBUG root
(appInterface.py:119) - taskId 27996832412
2015-11-10T00:56:40.613Z [0x7f98d86af700] DEBUG css.CssAccess
(core/modules/css/CssAccess.cc:95) - Create CSS instance from config map
2015-11-10T00:56:40.613Z [0x7f98d86af700] DEBUG css.CssAccess
(core/modules/css/CssAccess.cc:148) - Create CSS instance with mysql store
________________________________
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
________________________________
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
