Hello Daniel,
Thanks for this answer, i understand very well your arguments and i
agree with them.
Nevertheless, here's some more details which i ommitted in my first
email and which may help in defining a good solution :
Here's the *only* file which is needed to implement the custom.py
solution in eupspkg (with TaP feature), neither additional code is
required :
/eups@clrlsst-dbmaster-vm:~/eupspkg/contrib $ cat
qserv/patches/ldependencies_paths.patch//
//diff -rupN qserv.orig/core/custom.py qserv/core/custom.py//
//--- qserv.orig/core/custom.py 1970-01-01 01:00:00.000000000 +0100//
//+++ qserv/core/custom.py 2014-02-20 09:35:43.514467583 +0100//
//@@ -0,0 +1,17 @@//
//+import os//
//+import distutils.sysconfig as ds//
//+//
//+XROOTD_DIR=os.getenv("XROOTD_DIR")//
//+MYSQL_DIR=os.getenv("MYSQL_DIR")//
//+PROTOBUF_DIR=os.getenv("PROTOBUF_DIR")//
//+PYTHONPATH=os.getenv("PYTHONPATH")//
//+//
//+//
//+XROOTD_INC = os.path.join(XROOTD_DIR, "include","xrootd")//
//+MYSQL_INC = os.path.join(MYSQL_DIR,"include")//
//+//
//+MYSQL_LIB = os.path.join(MYSQL_DIR,"lib","mysql")//
//+XROOTD_LIB = os.path.join(XROOTD_DIR,"lib64")//
//+PROTOBUF_LIB = os.path.join(PROTOBUF_DIR,"lib")//
//+//
//+PROTOC=os.path.join(PROTOBUF_DIR,"bin","protoc")/
(please note that qserv/ doesn't have to be the qserv source directory,
but only the directory containing qserv packaging for eups, and qserv
tarball)
Furthermore, with this solution, it would be possible to use the default
eupspkg procedure (which launch "scons" by default) and so to remove the
qserv/ups/eupspkh.sh configuration file from the packaging.
Whereas, if we needed to launch "scons --eups", we would have to
overload build() and install() functions in "qserv/ups/eupspkh.sh"
So the custom.py solution would lead to less code in qserv build system,
and wouldn't add complexity in eupspkg packaging, except a small and
understable patch file. This could satisfying in the short-term, don't
you think so ?
Furthermore, eupspkg packages for Qserv have currently been reviewed here :
https://dev.lsstcorp.org/trac/ticket/3149
and please note that K.T. made several times this remark about Mario and I eupspkg packaging :
*/Is it safe to depend on /**/eups/**/using
/**/{PKGNAME}_DIR/**/environment variables?/*
Nevertheless, in my understanding, both the "custom.py" and the "scons
--eups" (cf. l. 65-66 in core/site_scons/eupsLib.py ) relies on
*/{PKGNAME}_DIR /*env variables.
I think that only sconsUtils may use an other technique to retrieve
dependencies path. So if we want not to rely on {PKGNAME}_DIR path in
the future, we may have to switch to sconsUtils.
This could require complex updates of Qserv SConstruct file.
That's why i would propose next solution :
- in the short-term rely on "custom.py" technique, which is simple and
straightforward,
- in the mid-term, if we're sure we mustn't use {PKGNAME}_DIR, develop a
solution relying on sconsUtils. On the other hand, if we know we can
rely on {PKGNAME}_DIR, we could improve the first technique, by using
"scons --eups" for example.
I hope these proposals are worth being studied.
Have a nice day,
Fabrice
On 02/19/2014 08:43 PM, Daniel L. Wang wrote:
> Hi Fabrice,
>
>> Qserv build ran successfully with eupspkg installed dependencies using
>> the "custom.py" configuration file.
>> But it doesn't seem to work while running command "scons --eups", or
>> "scons eups=1".
> It shouldn't, because I don't know enough of what I can derive from
> eups and how. The code is not implemented. If you show me the code you
> wrote to generate your custom.py, maybe I can finish it?
>
>> That's why, i propose you to use the "custom.py" configuration in the
>> eupspkg packaging.
> This is not what the custom.py was intended to address...
>
>> Here's why i propose this :
>>
>> - the new TaP (Tarball and patch) eupspkg feature allow to VERY easily
>> patch custom.py file. I can create this packaging in a few minutes,
> I agree. I'm sure that feature could do much more...
>
>> - having only one build process interface would be easier to maintain
>> and document,
> Ok.
>
>> - we won't have to modify the qserv build code if we stop using eupspkg,
> But we'll have to modify and write new code somewhere. Wherever it is,
> it will need to be tracked and kept up to date.
>
>> - we won't have to modify the qserv build code if eups interface change
>> (for exemple the name of the env variables),
> But we'll have to modify the code that writes the custom.py file. I
> would rather cut out that step. Why not be more direct?
>
>> - the build process interface would be the same for a stand-alone build
>> or a eups build, which would allow better debugging and understanding
>> for sysadmins,
> This is somewhat true. I wanted more integration for eups building,
> and I don't want to use files as a way of transmitting information
> from one part of qserv to another.
>
>> - the "custom.py" technique could be also used in rpm and deb packaging,
>> and so their would be only one interface to maintain for all packaging
>> system,
> One interface, but it's more generality than is needed. 'custom.py'
> really is a way to insert arbitrary python code into the build system.
> Each of the other packaging systems requires some integration logic,
> and usually it is fragile because the people performing the packaging
> are usually not part of the development team of the software that is
> being packaged. In our case, we are one team writing the software and
> doing the packaging. I think there is less code to manage overall if
> it is more integrated.
>
>> - none of all the qserv dependencies, or stack dependencies, contain
>> eups specific code in their build scripts,
> No, but there are eups-specific files there. The eups-specific plugin
> I propose is isolated.
>> - it is less job for us, and leads to a simpler solution,
> I don't think so. You'll need to write the integration logic
> somewhere--whether it's bash/perl/python code to generate the right
> command line to a './configure' script or a build-configuration file
> or a plugin into the build system. It's just a matter of where it goes.
>
>> Would you agree with it ?
>> Do you think i miss some important point ?
> I still need some convincing. One of the things I hate about qserv is
> that there are so many files that are created for passing information
> between its pieces. We have separate config files for the master/czar,
> proxy(sort of), my.cnf(worker AND master), and xrootd/cmsd. And in
> some way, we are asking the user to keep them consistent. Why? We
> control the code that generates every single one of them. Since we are
> writing the code that generates them, why do the generation only once
> during installation? If we can do the generation just-in-time, or not
> at all (maybe we can communicate the information directly!), we don't
> have to worry about those files being out of sync.
>
> If you insist, you might be able to convince me of a "pkgconfig.py"
> file that mostly works like custom.py, but I'd really rather not.
>
> What do you think? I may have missed something as well.
>
> -Daniel
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
