Hello Daniel,
Thanks for your answer and for your help,
My comments are below :
On 01/10/2014 03:22 AM, Daniel L. Wang wrote:
> Hi Fabrice,
>
> Sorry again about the delay in responding.
>
> common/ and master/ are obsolete, so they shouldn't build. Before I
> merge, I need to remove them and make sure nothing else is needed from
> them.
is worker also obsolete ?
>
> The custom.py is an attempt to reduce/eliminate the use of environment
> variables for building. So far, my custom.py looks like:
> -----
> MYSQL_LIB="/u1/local/lib/mysql"
> GEOMETRY="/u1/lsst/qserv/master/geometry.py"
> PYTHONPATH="/u1/local/lib/python2.5/site-packages"
> SYS_PYTHONPATH="/usr/lib64/python2.6/site-packages"
> ----
> But I would like to work with you to figure out exactly what needs to
> be specified. So far: mysql include/lib, xrootd include/lib, protobufs
> include/lib/protoc. Ideally these should always point at *installed*
> paths, rather than source directories. Boost normally gets detected
> in the system paths. Are you installing boost via eups? If so, maybe
> there should be separate lines for boost include/lib paths in custom.py.
i don't install eups via boost for now.
>
> Also, if EUPS sets up the linker paths and include paths in
> environment variables, we can pull those in from custom.py with
> something like:
> import os
> LD_LIBRARY_PATH=os.getenv("LD_LIBRARY_PATH")
> ...
>
At the moment i'm having a look at sconsutils which isn't documented but
which allow to manage automatically compile flags (for example CCPPATH
and LIBPATH) in scons scripts.
Using it would need a big effort to understand how it works, but would
allow qserv build process to be more LSST-compliant.
Furthermore it would certainly ease add of new dependencies and reduce
the size of Qserv Scons(truct,cript) files.
I sent an email to K.T. in order he decide if use of sconsUtils is
required or not. I propose to wait for its answer before defining the
solution for setting up build deps ?
The custom.py solution works and it's simple, but K.T. may decide to
rely on a more LSST compliant solution (like for eups integration in
build procedure).
>
> But let's work together on figuring out the best way to apply the
> paths from eups. Can you think of any other packages with paths that
> need to be setup to build within core/?
Here's my custom.py, i use a mix of the two techniques you proposed above :
fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/modules1_witheups) $
cat core/custom.py
import os
import distutils.sysconfig as ds
MYSQL_LIB=os.path.join(os.getenv("MYSQL_DIR"),"lib","mysql")
XROOTD_LIB=os.path.join(os.getenv("XROOTD_DIR"),"lib64")
PROTOBUF_LIB=os.path.join(os.getenv("PROTOBUF_DIR"),"lib")
QSERV_PYTHONPATH=os.path.join(os.getenv("PYTHON_DIR"),"lib","python"+ds.get_python_version(),"site-packages")
I add to hack core/modules/SConscript
(cf. git diff u/fjammes/modules1_witheups u/danielw/modules1) to
initialize CPPPATH (pat to include files) and then to call scons like that :
scons CPPPATH="${XROOTD_DIR}/include/xrootd:${MYSQL_DIR}/include"
Is there a better way to retrieve path to include files for building
core please ?
Have a nice day and W.E.
Fabrice
>
> Would it make your life easier to allow a configuration option like
> "--prefix=/usr/local" where the targets of core/ are "installed" into?
> Right now, core/ builds into core/bld/ , and it places the
> "installation" in core/bld/dist . You will find the python imports in
> core/bld/dist/lsst. Can you think of a better way? I would like to
> discuss this with you so we can determine the best solution that can
> integrate with eups and be as obvious as possible for someone to
> package a qserv rpm or install qserv without the help of eups. I
> would love for someone to be able to install qserv with "yum install
> qserv" or "aptitude install qserv".
>
>
> Let me know what you think,
> -Daniel
>
>
>
> On 01/09/2014 02:18 PM, Fabrice Jammes wrote:
>> Hello Daniel,
>>
>> First of all i wish you a merry christmas and a happy new year ;-).
>> I wish you all the best for 2014.
>>
>> Currently, i try to build your Qserv branch after having installed
>> all qserv deps (but not scisql), with eups (cf.
>> https://github.com/fjammes/misc, tag v0.5).
>> First i forked latest version of u/danielw/modules1 in
>> u/fjammes/modules1_witheups. You can have a look at the modifications
>> i've done in this branch.
>> (git diff remotes/origin/u/danielw/modules1 u/fjammes/modules1_witheups)
>> Please note that new scons.sh files only contains notes and comments
>> about building code in the same directory but are not intended to be
>> executed in their current state.
>>
>> As you know the build procedure isn't documented yet, so i try to do
>> some reverse engineering.
>> Now i understand better how it works, but i have certainly
>> misunderstood some pointsn that's why i would need your knowledge in
>> order to use it in the right way.
>>
>> *1. in core/ directory,* i can build successfully using next commands :
>>
>> # load eups config
>> cd ~/src/misc/eups/; source env.sh; cd -
>> # set up environment for eups product below
>> setup mysql
>> setup xrootd
>> setup protobuf
>> # TODO each python package must deploy locally its eggs
>> setup python
>> # here MYSQL_DIR contains mysql install dir, same for XROOTD_DIR and
>> others.
>>
>> # next env variables are loaded by scons procedure
>> (cf.detectProtobufs())
>> export PROTOC=${PROTOBUF_DIR}/bin/protoc
>> export PROTOC_INC=${PROTOBUF_DIR}/include
>> export PROTOC_LIB=${PROTOBUF_DIR}/lib
>>
>> # a custom.py file as also been added in order to set LIBPATH for
>> mysql, xrootd and protobuf, and PYTHONPATH
>>
>> # i've hacked core/modules/SConscript in order to set CPPPATH for
>> scons arguments, so that scons can find include files for xrootd and
>> mysql
>> scons CPPPATH="${XROOTD_DIR}/include/xrootd:${MYSQL_DIR}/include"
>> ...
>> g++ -o bld/libxrdoss.so -shared bld/util/Substitution.os
>> bld/mysql/MySqlConnection.os bld/wconfig/Config.os
>> bld/util/SqlConnection.os bld/wbase/Base.os
>> bld/obsolete/QservPathStructure.os bld/util/MmapFile.os
>> bld/util/SqlResults.os bld/util/WorkQueue.os bld/log/Logger.os
>> bld/util/Thread.os bld/obsolete/QservPath.os bld/xrdoss/QservOss.os
>> bld/mysql/SqlConfig.os bld/util/xrootd.os bld/util/SqlErrorObject.os
>> bld/wpublish/MySqlExportMgr.os bld/util/SqlFragmenter.os
>> bld/util/Timer.os -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql
>> -L/opt/lsst/Linux64/mysql/5.1.61/lib/mysql
>> -L/opt/lsst/Linux64/protobuf/2.4.1/lib
>> -L/opt/lsst/Linux64/xrootd/qs5/lib64 -L/usr/lib -lboost_regex-mt
>> -lboost_signals-mt -lboost_thread-mt -lmysqlclient_r -lssl -lcrypto
>> Install file: "bld/libxrdoss.so" as "bld/dist/lib/libxrdoss.so"
>> Install file: "bld/_masterLib.so" as
>> "bld/dist/lsst/qserv/master/_masterLib.so"
>> *scons: done building targets.*
>>
>>
>> # i've hacked core/site_scons/detect.py in order to detect boost
>> include files
>>
>> Is it possible to build without above hacks ? Could we use the same
>> method for loading all kind of build parameters (i.e. PRODUCT_DIR,
>> LIBPATH, and CPPPATH ) ?
>> Here, maybe K.T. will impose the use of sconsutils, i think this tool
>> may do the job.
>>
>> *
>> **2. i also tried to build common/ :*
>>
>> export MYSQL_ROOT=${MYSQL_DIR}
>> export PROTOC=${PROTOBUF_DIR}/bin/protoc
>> export PROTOC_INC=${PROTOBUF_DIR}/include
>> export PROTOC_LIB=${PROTOBUF_DIR}/lib
>>
>> eups@clrlsst-dbmaster-vm:~/src/qserv/common
>> (u/fjammes/modules1_witheups) $ rm -rf .scon*
>> eups@clrlsst-dbmaster-vm:~/src/qserv/common
>> (u/fjammes/modules1_witheups) $ scons
>> scons: Reading SConscript files ...
>> Protocol buffers using
>> protoc=/opt/lsst/Linux64/protobuf/2.4.1/bin/protoc with
>> include=/opt/lsst/Linux64/protobuf/2.4.1/include and
>> lib=/opt/lsst/Linux64/protobuf/2.4.1/lib
>> debug is 0
>> Checking for C++ header file boost/format.hpp... yes
>> Checking for C++ header file boost/thread.hpp... yes
>> Checking for C++ library mysqlclient_r... yes
>> Checking whether mysql_next_result is declared... yes
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> scons: *** [include/lsst/qserv/QservPath.hh] Source
>> `src/QservPath.hh' not found, needed by target
>> `include/lsst/qserv/QservPath.hh'.
>> scons: building terminated because of errors.
>>
>> Could we here also unify the method for loading build parameters, and
>> try to use the same that in 1., if it is possible ?
>> I think that common should build qserv_common and qserv_proto which
>> are required by master and worker.
>> Could you help me please ?
>>
>>
>> *3. i also tried to build master/*
>>
>> Here's what i've done :
>>
>> setup protobuf
>> setup xrootd
>> setup mysql
>> export SEARCH_ROOTS="${XROOTD_DIR}:${MYSQL_DIR}"
>>
>> And here's what i get, errors are in bold :
>>
>> eups@clrlsst-dbmaster-vm:~/src/qserv/master
>> (u/fjammes/modules1_witheups) $ scons
>> scons: Reading SConscript files ...
>> Using xrootd inc/lib: /opt/lsst/Linux64/xrootd/qs5/include/xrootd
>> /opt/lsst/Linux64/xrootd/qs5/lib64
>> *Checking for C++ library qserv_common... no*
>> Could not find qserv_common lib
>> *Checking for C++ library qserv_proto... no*
>> Could not find qserv_proto lib
>> Checking for C++ library boost_thread-gcc34-mt... no
>> Checking for C++ library boost_thread-gcc41-mt... no
>> Checking for C++ library boost_thread-mt... yes
>> Checking for C++ library boost_regex-gcc34-mt... no
>> Checking for C++ library boost_regex-gcc41-mt... no
>> Checking for C++ library boost_regex-mt... yes
>> Checking for C++ library antlr... yes
>> DEBUG <SCons.SConf.SConfBase object at 0x1577f10>
>> Checking for C library ssl... yes
>> Checking for C library crypto... yes
>> Checking for C++ library mysqlclient_r... yes
>> Checking for C++ library XrdUtils... yes
>> Checking for C++ library XrdClient... yes
>> Checking for C++ library XrdPosix... yes
>> Checking for C++ library XrdPosixPreload... yes
>> Checking for C++ header file XrdPosix/XrdPosixLinkage.hh... yes
>> *IOError: [Errno 2] No such file or directory: 'bin/myWrapper':*
>> File "/home/eups/src/qserv/master/SConstruct", line 318:
>> SConscript("SConscript.runner", exports='env') # Build a runner
>> script
>> File "/usr/lib/scons/SCons/Script/SConscript.py", line 614:
>> return method(*args, **kw)
>> File "/usr/lib/scons/SCons/Script/SConscript.py", line 551:
>> return _SConscript(self.fs, *files, **subst_kw)
>> File "/usr/lib/scons/SCons/Script/SConscript.py", line 260:
>> exec _file_ in call_stack[-1].globals
>> File "/home/eups/src/qserv/master/SConscript.runner", line 56:
>> makeRunner("bin/myWrapper")
>> File "/home/eups/src/qserv/master/SConscript.runner", line 48:
>> open(target, "w").write(s)
>>
>> Could you help me please ?
>>
>>
>> Thanks in advance for your help and have a nice day,
>>
>> Fabrice
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|