Hi Fabrice,

Sorry again about the delay in responding.

common/ and master/ are obsolete, so they shouldn't build. Before I 
merge, I need to remove them and make sure nothing else is needed from them.

The custom.py is an attempt to reduce/eliminate the use of environment 
variables for building. So far, my custom.py looks like:
-----
MYSQL_LIB="/u1/local/lib/mysql"
GEOMETRY="/u1/lsst/qserv/master/geometry.py"
PYTHONPATH="/u1/local/lib/python2.5/site-packages"
SYS_PYTHONPATH="/usr/lib64/python2.6/site-packages"
----
But I would like to work with you to figure out exactly what needs to be 
specified. So far: mysql include/lib, xrootd include/lib, protobufs 
include/lib/protoc.  Ideally these should always point at *installed* 
paths, rather than source directories.  Boost normally gets detected in 
the system paths. Are you installing boost via eups? If so, maybe there 
should be separate lines for boost include/lib paths in custom.py.

Also, if EUPS sets up the linker paths and include paths in environment 
variables, we can pull those in from custom.py with something like:
import os
LD_LIBRARY_PATH=os.getenv("LD_LIBRARY_PATH")
...


But let's work together on figuring out the best way to apply the paths 
from eups. Can you think of any other packages with paths that need to 
be setup to build within core/?

Would it make your life easier to allow a configuration option like 
"--prefix=/usr/local" where the targets of core/ are "installed" into? 
Right now, core/ builds into core/bld/ , and it places the 
"installation" in core/bld/dist . You will find the python imports in 
core/bld/dist/lsst. Can you think of a better way? I would like to 
discuss this with you so we can determine the best solution that can 
integrate with eups and be as obvious as possible for someone to package 
a qserv rpm or install qserv without the help of eups.  I would love for 
someone to be able to install qserv with "yum install qserv" or 
"aptitude install qserv".


Let me know what you think,
-Daniel



On 01/09/2014 02:18 PM, Fabrice Jammes wrote:
> Hello Daniel,
>
> First of all i wish you a merry christmas and a happy new year ;-).
> I wish you all the best for 2014.
>
> Currently, i try to build your Qserv branch after having installed all 
> qserv deps (but not scisql), with eups (cf. 
> https://github.com/fjammes/misc, tag v0.5).
> First i forked latest version of  u/danielw/modules1 in 
> u/fjammes/modules1_witheups. You can have a look at the modifications 
> i've done in this branch.
> (git diff remotes/origin/u/danielw/modules1 u/fjammes/modules1_witheups)
> Please note that new scons.sh files only contains notes and comments 
> about building code in the same directory but are not intended to be 
> executed in their current state.
>
> As you know the build procedure isn't documented yet, so i try to do 
> some reverse engineering.
> Now i understand better how it works, but  i have certainly 
> misunderstood some pointsn that's why i would need your knowledge in 
> order to use it in the right way.
>
> *1. in core/ directory,* i can build successfully using next commands :
>
> # load eups config
>  cd ~/src/misc/eups/; source env.sh; cd -
> # set up environment for eups product below
> setup mysql
> setup xrootd
> setup protobuf
> # TODO each python package must deploy locally its eggs
> setup python
> # here MYSQL_DIR contains mysql install dir, same for XROOTD_DIR and 
> others.
>
> # next env variables are loaded by scons procedure (cf.detectProtobufs())
> export PROTOC=${PROTOBUF_DIR}/bin/protoc
> export PROTOC_INC=${PROTOBUF_DIR}/include
> export PROTOC_LIB=${PROTOBUF_DIR}/lib
>
> # a custom.py file as also been added in order to set LIBPATH for 
> mysql, xrootd and protobuf, and PYTHONPATH
>
> # i've hacked core/modules/SConscript in order to set CPPPATH for 
> scons arguments, so that scons can find include files for xrootd and mysql
> scons CPPPATH="${XROOTD_DIR}/include/xrootd:${MYSQL_DIR}/include"
> ...
> g++ -o bld/libxrdoss.so -shared bld/util/Substitution.os 
> bld/mysql/MySqlConnection.os bld/wconfig/Config.os 
> bld/util/SqlConnection.os bld/wbase/Base.os 
> bld/obsolete/QservPathStructure.os bld/util/MmapFile.os 
> bld/util/SqlResults.os bld/util/WorkQueue.os bld/log/Logger.os 
> bld/util/Thread.os bld/obsolete/QservPath.os bld/xrdoss/QservOss.os 
> bld/mysql/SqlConfig.os bld/util/xrootd.os bld/util/SqlErrorObject.os 
> bld/wpublish/MySqlExportMgr.os bld/util/SqlFragmenter.os 
> bld/util/Timer.os -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql 
> -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql 
> -L/opt/lsst/Linux64/protobuf/2.4.1/lib 
> -L/opt/lsst/Linux64/xrootd/qs5/lib64 -L/usr/lib -lboost_regex-mt 
> -lboost_signals-mt -lboost_thread-mt -lmysqlclient_r -lssl -lcrypto
> Install file: "bld/libxrdoss.so" as "bld/dist/lib/libxrdoss.so"
> Install file: "bld/_masterLib.so" as 
> "bld/dist/lsst/qserv/master/_masterLib.so"
> *scons: done building targets.*
>
>
> # i've hacked core/site_scons/detect.py in order to detect boost 
> include files
>
> Is it possible to build without above hacks ? Could we use the same 
> method for loading all kind of build parameters (i.e. PRODUCT_DIR, 
> LIBPATH, and CPPPATH ) ?
> Here, maybe K.T. will impose the use of sconsutils, i think this tool 
> may do the job.
>
> *
> **2. i also tried to build common/ :*
>
> export MYSQL_ROOT=${MYSQL_DIR}
> export PROTOC=${PROTOBUF_DIR}/bin/protoc
> export PROTOC_INC=${PROTOBUF_DIR}/include
> export PROTOC_LIB=${PROTOBUF_DIR}/lib
>
> eups@clrlsst-dbmaster-vm:~/src/qserv/common 
> (u/fjammes/modules1_witheups) $ rm -rf .scon*
> eups@clrlsst-dbmaster-vm:~/src/qserv/common 
> (u/fjammes/modules1_witheups) $ scons
> scons: Reading SConscript files ...
> Protocol buffers using 
> protoc=/opt/lsst/Linux64/protobuf/2.4.1/bin/protoc with 
> include=/opt/lsst/Linux64/protobuf/2.4.1/include and 
> lib=/opt/lsst/Linux64/protobuf/2.4.1/lib
> debug is 0
> Checking for C++ header file boost/format.hpp... yes
> Checking for C++ header file boost/thread.hpp... yes
> Checking for C++ library mysqlclient_r... yes
> Checking whether mysql_next_result is declared... yes
> scons: done reading SConscript files.
> scons: Building targets ...
> scons: *** [include/lsst/qserv/QservPath.hh] Source `src/QservPath.hh' 
> not found, needed by target `include/lsst/qserv/QservPath.hh'.
> scons: building terminated because of errors.
>
> Could we here also unify the method for loading build parameters, and 
> try to use the same that in 1., if it is possible ?
> I think that common should build qserv_common and qserv_proto which 
> are required by master and worker.
> Could you help me please ?
>
>
> *3. i also tried to build master/*
>
> Here's what i've done :
>
> setup protobuf
> setup xrootd
> setup mysql
> export SEARCH_ROOTS="${XROOTD_DIR}:${MYSQL_DIR}"
>
> And here's what i get, errors are in bold :
>
> eups@clrlsst-dbmaster-vm:~/src/qserv/master 
> (u/fjammes/modules1_witheups) $ scons
> scons: Reading SConscript files ...
> Using xrootd inc/lib:  /opt/lsst/Linux64/xrootd/qs5/include/xrootd 
> /opt/lsst/Linux64/xrootd/qs5/lib64
> *Checking for C++ library qserv_common... no*
> Could not find qserv_common lib
> *Checking for C++ library qserv_proto... no*
> Could not find qserv_proto lib
> Checking for C++ library boost_thread-gcc34-mt... no
> Checking for C++ library boost_thread-gcc41-mt... no
> Checking for C++ library boost_thread-mt... yes
> Checking for C++ library boost_regex-gcc34-mt... no
> Checking for C++ library boost_regex-gcc41-mt... no
> Checking for C++ library boost_regex-mt... yes
> Checking for C++ library antlr... yes
> DEBUG <SCons.SConf.SConfBase object at 0x1577f10>
> Checking for C library ssl... yes
> Checking for C library crypto... yes
> Checking for C++ library mysqlclient_r... yes
> Checking for C++ library XrdUtils... yes
> Checking for C++ library XrdClient... yes
> Checking for C++ library XrdPosix... yes
> Checking for C++ library XrdPosixPreload... yes
> Checking for C++ header file XrdPosix/XrdPosixLinkage.hh... yes
> *IOError: [Errno 2] No such file or directory: 'bin/myWrapper':*
>   File "/home/eups/src/qserv/master/SConstruct", line 318:
>     SConscript("SConscript.runner", exports='env') # Build a runner script
>   File "/usr/lib/scons/SCons/Script/SConscript.py", line 614:
>     return method(*args, **kw)
>   File "/usr/lib/scons/SCons/Script/SConscript.py", line 551:
>     return _SConscript(self.fs, *files, **subst_kw)
>   File "/usr/lib/scons/SCons/Script/SConscript.py", line 260:
>     exec _file_ in call_stack[-1].globals
>   File "/home/eups/src/qserv/master/SConscript.runner", line 56:
>     makeRunner("bin/myWrapper")
>   File "/home/eups/src/qserv/master/SConscript.runner", line 48:
>     open(target, "w").write(s)
>
> Could you help me please ?
>
>
> Thanks in advance for your help and have a nice day,
>
> Fabrice
>
>
>
>
> ------------------------------------------------------------------------
>
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1