Print

Print
Hello Daniel,

First of all i wish you a merry christmas and a happy new year ;-).
I wish you all the best for 2014.

Currently, i try to build your Qserv branch after having installed all qserv deps (but not scisql), with eups (cf.  https://github.com/fjammes/misc, tag v0.5).
First i forked latest version of  u/danielw/modules1 in u/fjammes/modules1_witheups. You can have a look at the modifications i've done in this branch.
(git diff remotes/origin/u/danielw/modules1 u/fjammes/modules1_witheups)
Please note that new scons.sh files only contains notes and comments about building code in the same directory but are not intended to be executed in their current state.

As you know the build procedure isn't documented yet, so i try to do some reverse engineering.
Now i understand better how it works, but  i have certainly misunderstood some pointsn that's why i would need your knowledge in order to use it in the right way.

1. in core/ directory, i can build successfully using next commands :

# load eups config
 cd ~/src/misc/eups/; source env.sh; cd -
# set up environment for eups product below
setup mysql
setup xrootd
setup protobuf
# TODO each python package must deploy locally its eggs
setup python
# here MYSQL_DIR contains mysql install dir, same for XROOTD_DIR and others.

# next env variables are loaded by scons procedure (cf.detectProtobufs())
export PROTOC=${PROTOBUF_DIR}/bin/protoc
export PROTOC_INC=${PROTOBUF_DIR}/include
export PROTOC_LIB=${PROTOBUF_DIR}/lib

# a custom.py file as also been added in order to set LIBPATH for mysql, xrootd and protobuf, and PYTHONPATH

# i've hacked core/modules/SConscript in order to set CPPPATH for scons arguments, so that scons can find include files for xrootd and mysql
scons CPPPATH="${XROOTD_DIR}/include/xrootd:${MYSQL_DIR}/include"
...
g++ -o bld/libxrdoss.so -shared bld/util/Substitution.os bld/mysql/MySqlConnection.os bld/wconfig/Config.os bld/util/SqlConnection.os bld/wbase/Base.os bld/obsolete/QservPathStructure.os bld/util/MmapFile.os bld/util/SqlResults.os bld/util/WorkQueue.os bld/log/Logger.os bld/util/Thread.os bld/obsolete/QservPath.os bld/xrdoss/QservOss.os bld/mysql/SqlConfig.os bld/util/xrootd.os bld/util/SqlErrorObject.os bld/wpublish/MySqlExportMgr.os bld/util/SqlFragmenter.os bld/util/Timer.os -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql -L/opt/lsst/Linux64/protobuf/2.4.1/lib -L/opt/lsst/Linux64/xrootd/qs5/lib64 -L/usr/lib -lboost_regex-mt -lboost_signals-mt -lboost_thread-mt -lmysqlclient_r -lssl -lcrypto
Install file: "bld/libxrdoss.so" as "bld/dist/lib/libxrdoss.so"
Install file: "bld/_masterLib.so" as "bld/dist/lsst/qserv/master/_masterLib.so"
scons: done building targets.


# i've hacked core/site_scons/detect.py in order to detect boost include files

Is it possible to build without above hacks ? Could we use the same method for loading all kind of build parameters (i.e. PRODUCT_DIR, LIBPATH, and CPPPATH ) ?
Here, maybe K.T. will impose the use of sconsutils, i think this tool may do the job.


2. i also tried to build common/ :

export MYSQL_ROOT=${MYSQL_DIR}
export PROTOC=${PROTOBUF_DIR}/bin/protoc
export PROTOC_INC=${PROTOBUF_DIR}/include
export PROTOC_LIB=${PROTOBUF_DIR}/lib

eups@clrlsst-dbmaster-vm:~/src/qserv/common (u/fjammes/modules1_witheups) $ rm -rf .scon*
eups@clrlsst-dbmaster-vm:~/src/qserv/common (u/fjammes/modules1_witheups) $ scons
scons: Reading SConscript files ...
Protocol buffers using protoc=/opt/lsst/Linux64/protobuf/2.4.1/bin/protoc with include=/opt/lsst/Linux64/protobuf/2.4.1/include and lib=/opt/lsst/Linux64/protobuf/2.4.1/lib
debug is 0
Checking for C++ header file boost/format.hpp... yes
Checking for C++ header file boost/thread.hpp... yes
Checking for C++ library mysqlclient_r... yes
Checking whether mysql_next_result is declared... yes
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [include/lsst/qserv/QservPath.hh] Source `src/QservPath.hh' not found, needed by target `include/lsst/qserv/QservPath.hh'.
scons: building terminated because of errors.

Could we here also unify the method for loading build parameters, and try to use the same that in 1., if it is possible ?
I think that common should build qserv_common and qserv_proto which are required by master and worker.
Could you help me please ?


3. i also tried to build master/

Here's what i've done :

setup protobuf
setup xrootd
setup mysql
export SEARCH_ROOTS="${XROOTD_DIR}:${MYSQL_DIR}"

And here's what i get, errors are in bold :

eups@clrlsst-dbmaster-vm:~/src/qserv/master (u/fjammes/modules1_witheups) $ scons
scons: Reading SConscript files ...
Using xrootd inc/lib:  /opt/lsst/Linux64/xrootd/qs5/include/xrootd /opt/lsst/Linux64/xrootd/qs5/lib64
Checking for C++ library qserv_common... no
Could not find qserv_common lib
Checking for C++ library qserv_proto... no
Could not find qserv_proto lib
Checking for C++ library boost_thread-gcc34-mt... no
Checking for C++ library boost_thread-gcc41-mt... no
Checking for C++ library boost_thread-mt... yes
Checking for C++ library boost_regex-gcc34-mt... no
Checking for C++ library boost_regex-gcc41-mt... no
Checking for C++ library boost_regex-mt... yes
Checking for C++ library antlr... yes
DEBUG <SCons.SConf.SConfBase object at 0x1577f10>
Checking for C library ssl... yes
Checking for C library crypto... yes
Checking for C++ library mysqlclient_r... yes
Checking for C++ library XrdUtils... yes
Checking for C++ library XrdClient... yes
Checking for C++ library XrdPosix... yes
Checking for C++ library XrdPosixPreload... yes
Checking for C++ header file XrdPosix/XrdPosixLinkage.hh... yes
IOError: [Errno 2] No such file or directory: 'bin/myWrapper':
  File "/home/eups/src/qserv/master/SConstruct", line 318:
    SConscript("SConscript.runner", exports='env') # Build a runner script
  File "/usr/lib/scons/SCons/Script/SConscript.py", line 614:
    return method(*args, **kw)
  File "/usr/lib/scons/SCons/Script/SConscript.py", line 551:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/usr/lib/scons/SCons/Script/SConscript.py", line 260:
    exec _file_ in call_stack[-1].globals
  File "/home/eups/src/qserv/master/SConscript.runner", line 56:
    makeRunner("bin/myWrapper")
  File "/home/eups/src/qserv/master/SConscript.runner", line 48:
    open(target, "w").write(s)

Could you help me please ?


Thanks in advance for your help and have a nice day,

Fabrice




Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1