Print

Print
Hi Bill,

Good job and good news ! Your install now works. It's not so easy to set up a working qserv instance, isn't it ;-)

On 26/11/2013 21:13, Bill Chickering wrote:
[log in to unmask]" type="cite">
Hi Fabrice -

Okay, my mistake. I now have a working instance of master branch qserv--thanks!

I did encounter a couple of minor snags, however. Perhaps these are unique to my setup or due to errors on my part, but I mention them for posterity. . . 

As part of my installation, I migrated data from another qserv instance. To do this, I merely copied %QSERV_BASE_DIR%/var/lib/mysql from the preexisting instance to the new instance. Having migrating metadata, qserv now successfully starts upon issuing `qserv-start`.
Copying %QSERV_BASE_DIR%/var/lib/mysql also copy %QSERV_BASE_DIR%/var/lib/mysql/mysql which contains administrative informations,of whose privileges and access rights. This may cause the next problem on qservResult database.
This kind of data loading is mastered by Douglas.
[log in to unmask]" type="cite">

At this point query execution fails, however, due to insufficient permissions on the qservResult database (I see errors in the qserv log saying ''@'localhost' cannot access qservResult). To remove this obstacle, I connect to the qservResult database as root and issue the command `grant all privileges on *.* to ''@'localhost' with grant option;`. Certainly this is not ideal, but it solved my immediate issue.

At this point query execution still fails due to the absence of %QSERV_BASE_DIR%/etc/emptyChunks.txt. Copying this file over from the preexisting instance solves the problem. I now have a working instance of qserv.
The test procedure create %QSERV_BASE_DIR%/etc/emptyChunks.txt for each test dataset (cf. qservdataloader.py). In case you load an other dataset, you also have to load its meta (and not only the QMS ones), of whose %QSERV_BASE_DIR%/etc/emptyChunks.txt file. All this is managed by Douglas scripts for several dataset (cf. qserv-chunkload.py and qserv-admin.pl) and should be covered by the future unified dataloading procedure.
[log in to unmask]" type="cite">

While qserv seems to be essentially working correctly, here's what I encounter upon issuing `qserv-start` and `qserv-stop`:

-bash-4.1$ qserv-start
chown: invalid group: `bchick:bchick'
chown: invalid group: `bchick:bchick'
Starting mysqld:                                           [  OK  ]
Starting mysql-proxy:                                      [  OK  ]
Starting xrootd:                                           [  OK  ]
Starting qms:                                              [  OK  ]
Starting qserv-master:                                     [  OK  ]
-bash-4.1$ qserv-stop
Stopping mysqld:                                           [  OK  ]
Stopping mysql-proxy:                                      [FAILED]
Stopping xrootd:                                           [  OK  ]
Stopping qms:                                              [  OK  ]
Stopping qserv-master:                                     [  OK  ]
-bash-4.1$ 

Note that all qserv-related processes have indeed stopped after issuing `qserv-stop`.

Thanks for these very interesting informations which are worth opening a Trac ticket about init.d script.

The chown warning is issued by mysqld init.d script because you don't belongs to group bchick. This init.d script is nearly the same than the one of SL6.2 distribution. This problem is non blocking and can be easily solved.
For the mysql-proxy issue, i need a more detailed diagnostic in order to solve it. Could you please find interesting things in the log ? is mysql-proxy pid file always here ? ...
The services should also be stopped in reverse order.
I think it's important to have robusts init.d scripts, but this require putting some stress on them with  multiple user tests.

Thanks and have a nice day,

Fabrice

[log in to unmask]" type="cite">
Anyhow, I'm happy to have a running instance of the master branch! Thanks for your help!

Regards,
Bill

On Nov 26, 2013, at 12:21 AM, Fabrice Jammes <[log in to unmask]> wrote:

Hello Bill,

I can't see the script start_xrootd, start_qms, or start_qserv mentionned in https://dev.lsstcorp.org/trac/wiki/db/Qserv/InstallAuto
are you reading the latest version of this page ?

Indeed, start_* scripts have been replaced by more standard init.d script since #3013.

qserv-start and qserv-stop are still used as wrapper to start/stop all qserv services, but you've first to source qserv-env.sh, in order to enable these 2 aliases.
If you want to know more about Qserv services fine grained management, please have a look at : https://dev.lsstcorp.org/trac/wiki/db/Qserv/initd

Hope this help.

Fabrice



On 11/25/2013 10:55 PM, Bill Chickering wrote:
[log in to unmask]" type="cite">
Hi Fabrice -

Thanks for merging #3006. I've pulled the latest master branch and am attempting to build a single node instance of qserv from scratch. It appears that the dependencies were successfully built. Here's the tail of the output I observe at the terminal after running `scons` within the src directory:

DEBUG : {'XRD_DIR': '/u1/bchick/sandbox2/qserv-master/xrootd', 'PROTOC': '/u1/bchick/sandbox2/qserv-master/bin/protoc', 'PROTOC_INC': '/u1/bchick/sandbox2/qserv-master/include', 'QSW_DBSOCK': '/u1/bchick/sandbox2/qserv-master/var/lib/mysql/mysql.sock', 'MYSQL_ROOT': '/u1/bchick/sandbox2/qserv-master', 'SHLVL': '2', 'OLDPWD': '/u1/bchick/sandbox2/qserv-master/qserv/master', 'SEARCH_ROOTS': '/u1/bchick/sandbox2/qserv-master', 'PWD': '/u1/bchick/sandbox2/qserv-master/qserv/worker', 'XRD_PLATFORM': 'x86_64_linux_26_dbg', 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin', 'LD_LIBRARY_PATH': '/u1/bchick/sandbox2/qserv-master/lib:/usr/lib64:/u1/bchick/sandbox2/qserv-master/lib/mysql:/u1/bchick/sandbox2/qserv-master/lib:/usr/lib64:/u1/bchick/sandbox2/qserv-master/qserv/worker/../common/lib:/u1/bchick/sandbox2/qserv-master/qserv/worker/../common/lib:/u1/bchick/sandbox2/qserv-master/lib:/u1/bchick/sandbox2/qserv-master/lib/mysql:/u1/bchick/sandbox2/qserv-master/qserv/worker/bldNoXrd', '_': '/usr/bin/scons', 'PROTOC_LIB': '/u1/bchick/sandbox2/qserv-master/lib'}
g++ -o tests/testSanityCheck.o -c -g -DNO_XROOTD_FS -I/u1/bchick/sandbox2/qserv-master/include/xrootd -I/u1/bchick/sandbox2/qserv-master/include/xrootd -Iinclude -I/u1/bchick/src/qserv-master/common/include -I/u1/bchick/sandbox2/qserv-master/include tests/testSanityCheck.cc
g++ -o tests/testSanityCheck tests/testSanityCheck.o bldNoXrd/lib/libqserv_worker.so bldNoXrd/lib/libqserv_workerCmsd.so -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/sandbox2/qserv-master/lib/mysql -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/sandbox2/qserv-master/lib -L/u1/bchick/sandbox2/qserv-master/lib/mysql -LbldNoXrd -lssl -lcrypto -lboost_regex-mt -lboost_thread-mt -lboost_signals-mt -lprotobuf -lssl -lcrypto -lmysqlclient_r -lmysqlclient_r -lqserv_common -lqserv_proto -lXrdUtils -lXrdClient -lXrdServer -lXrdSys -lXrdOfs -lXrdSfs -lmysqlclient_r -lXrdSys -lXrdClient -lXrdUtils
run_test(["tests/.tests/testSanityCheck"], ["tests/testSanityCheck"])
DEBUG : {'XRD_DIR': '/u1/bchick/sandbox2/qserv-master/xrootd', 'PROTOC': '/u1/bchick/sandbox2/qserv-master/bin/protoc', 'PROTOC_INC': '/u1/bchick/sandbox2/qserv-master/include', 'QSW_DBSOCK': '/u1/bchick/sandbox2/qserv-master/var/lib/mysql/mysql.sock', 'MYSQL_ROOT': '/u1/bchick/sandbox2/qserv-master', 'SHLVL': '2', 'OLDPWD': '/u1/bchick/sandbox2/qserv-master/qserv/master', 'SEARCH_ROOTS': '/u1/bchick/sandbox2/qserv-master', 'PWD': '/u1/bchick/sandbox2/qserv-master/qserv/worker', 'XRD_PLATFORM': 'x86_64_linux_26_dbg', 'PATH': '/usr/local/bin:/opt/bin:/bin:/usr/bin', 'LD_LIBRARY_PATH': '/u1/bchick/sandbox2/qserv-master/lib:/usr/lib64:/u1/bchick/sandbox2/qserv-master/lib/mysql:/u1/bchick/sandbox2/qserv-master/lib:/usr/lib64:/u1/bchick/sandbox2/qserv-master/qserv/worker/../common/lib:/u1/bchick/sandbox2/qserv-master/qserv/worker/../common/lib:/u1/bchick/sandbox2/qserv-master/lib:/u1/bchick/sandbox2/qserv-master/lib/mysql:/u1/bchick/sandbox2/qserv-master/qserv/worker/bldNoXrd', '_': '/usr/bin/scons', 'PROTOC_LIB': '/u1/bchick/sandbox2/qserv-master/lib'}
g++ -o tests/PathStructure_1.o -c -g -DNO_XROOTD_FS -I/u1/bchick/sandbox2/qserv-master/include/xrootd -I/u1/bchick/sandbox2/qserv-master/include/xrootd -Iinclude -I/u1/bchick/src/qserv-master/common/include -I/u1/bchick/sandbox2/qserv-master/include tests/PathStructure_1.cc
g++ -o tests/PathStructure_1 tests/PathStructure_1.o src/QservPathStructure.o -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/sandbox2/qserv-master/lib/mysql -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/sandbox2/qserv-master/lib -L/u1/bchick/sandbox2/qserv-master/lib/mysql -LbldNoXrd
scons: done building targets.
Stopping mysqld:                                           [  OK  ]
scons: done building targets.


And here is the tail of INSTALL-2013-11-25-12:36:46.log:

make[1]: Leaving directory `/u1/bchick/sandbox2/qserv-master/build/mysql-proxy-0.8.2/doc'
make[1]: Entering directory `/u1/bchick/sandbox2/qserv-master/build/mysql-proxy-0.8.2'
make[2]: Entering directory `/u1/bchick/sandbox2/qserv-master/build/mysql-proxy-0.8.2'
make[2]: Nothing to be done for `install-exec-am'.
test -z "/u1/bchick/sandbox2/qserv-master/lib/pkgconfig" || /bin/mkdir -p "/u1/bchick/sandbox2/qserv-master/lib/pkgconfig"
 /usr/bin/install -c -m 644 'mysql-proxy.pc' '/u1/bchick/sandbox2/qserv-master/lib/pkgconfig/mysql-proxy.pc'
 /usr/bin/install -c -m 644 'mysql-chassis.pc' '/u1/bchick/sandbox2/qserv-master/lib/pkgconfig/mysql-chassis.pc'
make[2]: Leaving directory `/u1/bchick/sandbox2/qserv-master/build/mysql-proxy-0.8.2'
make[1]: Leaving directory `/u1/bchick/sandbox2/qserv-master/build/mysql-proxy-0.8.2'
QSERV DEPENDENCIES SUCCESSFULLY INSTALLED

Within the qserv install directory, I now see the start_mysqlproxy script. I do not, however, see start_xrootd, start_qms, or start_qserv. These scripts should have been created during the install, correct? Did I make mistake or skip a step? (btw, I have read https://dev.lsstcorp.org/trac/wiki/db/Qserv/InstallAuto and tried to follow the procedure outlined there).

Thanks,
Bill


On Nov 25, 2013, at 4:48 AM, Fabrice Jammes <[log in to unmask]> wrote:

Hello Bill,

#3006 is now merged to master.

Have fun!

Fabrice

On 11/25/2013 09:29 AM, Bill Chickering wrote:
Sounds good. Thanks, Fabrice.


On Nov 25, 2013, at 12:23 AM, Fabrice Jammes <[log in to unmask]> wrote:

Hello Bill,

Ticket #3006 solves this issue. It as just been reviewed on Friday, i'll merge it soon and let you know.

Fabrice


On 11/24/2013 11:57 PM, Bill Chickering wrote:
Hi Guys -

I'm trying to build the master branch qserv from scratch (this is the master before merging 3019 and 3020). Running into the same problem I mentioned to Douglas and Fabrice during the hackathon. I thought that Fabrice said a ticket was merged that resolved this issue, but evidently that's not the case.
Here's the error I encounter:

run_test(["tests/.tests/QuerySql_1"], ["tests/QuerySql_1"])
g++ -o tests/QueueTest.o -c -g -DNO_XROOTD_FS -I/u1/bchick/sandbox2/qserv-master/include/xrootd -I/u1/bchick/sandbox2/qserv-master/include/xrootd -Iinclude -I/u1/bchick/src/qserv-master/common/include -
I/u1/bchick/sandbox2/qserv-master/include tests/QueueTest.cc
g++ -o tests/QueueTest tests/QueueTest.o -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/sandbox2/qserv-master/lib/mysql -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick
/src/qserv-master/common/lib -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/sandbox2/qserv-master/lib -L/u1/bchick/sandbox2/qserv-master/lib/mysql -LbldNoXrd
run_test(["tests/.tests/QueueTest"], ["tests/QueueTest"])
g++ -o tests/testSanityCheck.o -c -g -DNO_XROOTD_FS -I/u1/bchick/sandbox2/qserv-master/include/xrootd -I/u1/bchick/sandbox2/qserv-master/include/xrootd -Iinclude -I/u1/bchick/src/qserv-master/common/inc
lude -I/u1/bchick/sandbox2/qserv-master/include tests/testSanityCheck.cc
g++ -o tests/testSanityCheck tests/testSanityCheck.o bldNoXrd/lib/libqserv_worker.so bldNoXrd/lib/libqserv_workerCmsd.so -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/sandbox2/qserv-m
aster/lib/mysql -L/u1/bchick/sandbox2/qserv-master/lib -L/usr/lib64 -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/src/qserv-master/common/lib -L/u1/bchick/sandbox2/qserv-master/lib -L/u1/bchick/
sandbox2/qserv-master/lib/mysql -LbldNoXrd -lssl -lcrypto -lboost_regex-mt -lboost_thread-mt -lboost_signals-mt -lprotobuf -lssl -lcrypto -lmysqlclient_r -lmysqlclient_r -lqserv_common -lqserv_proto -lX
rdUtils -lXrdClient -lXrdServer -lXrdSys -lXrdOfs -lXrdSfs -lmysqlclient_r -lXrdSys -lXrdClient -lXrdUtils
run_test(["tests/.tests/testSanityCheck"], ["tests/testSanityCheck"])
*** TEST FAILED *** tests/testSanityCheck 201
scons: *** [tests/.tests/testSanityCheck] Error 201
scons: building terminated because of errors.

Any suggestions?

Thanks,
Bill


On Nov 22, 2013, at 7:44 PM, Jacek Becla <[log in to unmask]> wrote:

Bill,

If you need any help with merging your tickets to the master,
please let me know.

The page I mentioned at the meeting that might help:

https://dev.lsstcorp.org/trac/wiki/GitDemoAndTutorial#LSSTgitworkflowandbranchmanagementpolicy

Thanks
Jacek
########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1


Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1



Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1




Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1