On 12/26/13 3:02 , Fabrice Jammes wrote:
>

Hi Fabrice,
	Happy holidays to you too, and see for some answers below!

> 
> Nevertheless, i still need some help on some small details, in order to
> converge towards the right direction.
> 
> *1. *At the moment, the build procedure relies on HSC build style (like
> it is what was recommended by Paul Price one month ago, cf.
> http://hsca.ipmu.jp:8080/question/128/more-information-about-eups-create-distrib/)
> It builds and installs nearly all Qserv source dependencies (but not
> scisql and Qserv itself) on both Ubuntu 13.04 and SL6.2.
> For information, i did not use nor lssteups or sconsutils (i only have
> the qserv package to build with scons, and Qserv only links with
> protobuf, xrootd and mysql).
> 

That sounds great! Is there a git repo you can point me to to take a look?

> - Do you confirm i have to move all these build script to pkgbuild right
> now ? Is pkgbuild stable enough ? If yes, does
> https://github.com/RobertLuptonTheGood/eups.git master branch support
> pkgbuild, or shall i use
> an other branch ?

I just got EupsPkg (formerly known as pkgbuild) into a state where it's
ready for review. It's right now at 'eupspkg' branch of my EUPS fork in
github:

	https://github.com/mjuric/eups/tree/eupspkg

I plan to make a pull request to Robert (and ask others for comments as
well) imminently, and barring huge architectural disagreements I think
it will get merged into EUPS proper soon (perhaps ~two weeks or so).

Given that, I think it makes sense to start transitioning your build
scripts to eupspkg. It hope it won't be hard -- I've written
documentation in the docstring of eupspkg module:

	https://github.com/mjuric/eups/blob/eupspkg/python/eups/distrib/eupspkg.py

, made examples for all LSST stack external dependencies:

	https://dev.lsstcorp.org/cgit/personal/mjuric/eupspkg/external/

, and for LSST code that relies on scons (via sconsUtils), eupspkg knows
what to do out of the box (i.e., there's no need to modify it at all or
supply a ./ups/eupspkg file).

If you point me to your existing builder scripts, I can help with
transforming them to eupspkg variants. My guess, based on doing that for
the rest of DM stack dependencies, is that it shouldn't take more than
~2-3 hours.

> - Do you think Qserv build should relies on sconsutils ?

I'm not familiar with the details, so I'll let K-T et al answer.

> - Do you thinks sconsutils may be compatible with rpm format ?

I think it should be possible to write a fully automated eupspkg-to-spec
converter. All information needed to obtain and build the source is in
the eupspkg scripts.

> - Do you confirm Qserv build musn't rely on lssteups (I didn't notice
> the use of this tool while i investigated HSC build scripts.) ?
> 

The plan is for lssteups to be superseded by eupspkg; therefore, it
shouldn't be used any more.

More comments below:
> For the next three points, i answer you between the lines below :
> 
> On 12/20/2013 07:55 PM, Kian-Tat Lim wrote:
>> Fabrice,
>>
>>> Maybe it would be better if eups always run with system python ?
>> 	In the not-too-distant future (maybe even in January or
>> February), I think we want to convert the LSST Stack to run with the
>> system Python in all cases, including eups.
>>
>>> Indeed for now it switch to eups-installed python once command
>>> "setup python" is performed.
>> 	"setup python" would go away.  But I'm not sure what the problem
>> is with switching Pythons.
> 
> *2. *The problem is met with next hypothesis :
> - eups is compatible with python version i, but not with python version
> j, and eups is used to install a software, named SOFT, which require
> python version j. (note that j can even be lower than i)
> During SOFT install, eups will install python version j, and in current
> eups version, it will itself switch to python version j, and then the
> whole install procedure may break. Indeed all call to eups may fails
> because of the use of python version j,
> so it may even be not possible to go back (unsetup command may fails).
> 
> That's why i propose that, during eups installation, all call to eups
> executables are forced to be done with python version i. Note that all
> other calls to python done by the build scripts will use environment
> python, i.e. python version j.
> 
> (For exemple, to use this solution, eups/bin/eups would be simply
> updated like during by eups install process :
> --- a/bin/eups
> +++ b/bin/eups
> @@ -3,4 +3,4 @@
>  #  #!/usr/bin/env python options
>  #
>  unset PYTHONSTARTUP
> -*python* -S $EUPS_DIR/bin/eups_impl.py "$@"
> +*/usr/bin/python* -S $EUPS_DIR/bin/eups_impl.py "$@"
> )
> 
> This solution, trivial to implement, would make eups more stable and
> more generic, don't you think so ?
> (I expose briefly this issue to Christian Arnault, french manager for
> LSST computing activities and author of CMT, the atlas build system, and
> he agreed with me.)
> 

It's a reasonable proposal (IMHO), except that one wouldn't want to
hardcode the python path as /usr/bin/python, but autodetect it during
the install of EUPS and use that.

There may be systems that don't require /usr/bin/python to be installed
(e.g., various *BSDs), so hardcoding the path would prevent a
non-privileged user from installing EUPS with their own python.

That said:

> Note that i met this issue while switching from system python to
> virtualenv python during Qserv build process (cf.
> http://hsca.ipmu.jp:8080/question/167/installing-virtualenv-with-eups/).

I think we need to understand why EUPS needs to run python with -S;
that's what's causing the issue with virtualenv.

Btw., are you sure it really does this? I just grepped through the
source and couldn't find any invocations with '-S'.

Could it be that it's virtualenv has a bug somwehere? It's a really
bizarre error, since traceback is a simple pure Python module.

More below:
>>>> 	Regarding its use with virtualenv, EUPS has been designed with
>>>> virtual-env like functionality in mind -- specifically, it manages the
>>>> PYTHON_PATH for you. So using the two together is likely to generate
>>>> conflicts (though it may not be impossible).
>>> One of the main advantage of using virtualenv is that install of
>>> python libraries via eups is completely transparent : you don't have
>>> to manage a dedicated PYTHONPATH in eups build scripts of python
>>> libraries.
>>> It also isolate eups-installed python and then prevent conflicts
>>> between system python user libraries and eups-installed python user
>>> libraries.
>>> Of course, if you prefer, i can set up build script to install each
>>> Qserv python lib in a dedicated directory and then add it to
>>> PYTHONPATH.
>> 	We already have the PYTHONPATH manipulation in the rest of the
>> LSST Stack, so I think it's best for now to continue with that.  In the
>> longer run, I would like to see all LSST Stack Python modules installed
>> into a single directory, almost certainly an LSST-specific one and not
>> the system site-packages -- but they would be swapped in and out on
>> setup of the individual packages through the use of symbolic links.
> *3. *This long-term solution looks fine. So i confirm i will implement
> PYTHONPATH manipulation in my next version of Qserv packaging tool.
>> 	One of the main reasons to use the system Python is to allow the
>> user to incorporate other packages into their usage of the LSST Stack.
>> virtualenv does not work well with this usage model.  This may not be an
>> issue with qserv, however, which is server-side and can insist on a more
>> rigorously-controlled environment.
>>
> *
> 4. *Qserv now relies on virtualenv but it also may works with system-python.
> 
> An interesting feature of virtualenv in a rigorously-controlled
> environment is that it may switch PYTHONPATH while its
> activated/deactivated.
> 

But that is *exactly* the problem that EUPS tries to solve, not just for
Python modules but for other code as well, and with much more granularity.

Think of EUPS as being an "uber-virtualenv", where you can choose to
activate or deactivate not just the entire environment at once, but each
*individual package*. The equivalent of 'activate' with EUPS is 'setup',
and 'deactivate' is 'unsetup'.

For example, say I have a Python code foo v1.2.3 that depends on a
Python code bar v1.0.0, a shared library named baz v1.1.1 (i.e.,
libbaz.so.1.1.1), and a command-line utility named qux 3.2.1:

    foo v1.2.3
    +-- bar v1.0.0
    +-- baz v1.1.1
    +-- qux v3.2.1

Say they're all installed in an EUPS stack. I can set them all up by
running:

    setup foo 1.2.3

This will bring foo and bar onto PYTHONPATH, baz onto LD_LIBRARY_PATH,
and qux onto PATH. Virtualenv's 'activate' would just do the first of
these (at least that was the case when I last used it). To do the
reverse, I'd run 'unsetup foo', and both PYTHONPATH and LD_LIBRARY_PATH
would both get properly cleaned up. That is what virtualenv's
'deactivate' does, but, again, just for Python modules.

Now, if I wanted to install a new version of foo (say v1.3.0), I can
easily do that with EUPS without overwriting the existing one. It won't
interfere with v1.2.3, and it will reuse the already installed bar and
baz (assuming it still depends on them). With virtualenv, I'd need to
build a whole new environment just to update this one package as v1.3.0
will want to overwrite the version already in the virtualenv container.
This is an issue when there may be 50+ other packages in the stack (and
with LSST, there are).

Similarly, if I next installed foo v1.4.0 that now requires bar v1.2.0,
I could do that easily with EUPS; neither will interfere with other
versions, and when I run:

    setup foo 1.4.0

EUPS will know it needs to also setup bar 1.2.0 (because it tracks
dependencies). Again, with virtualenv, I'd need to build a whole new
environment; it has no notion of internal product dependencies.

Finally, EUPS lets you "mix and match" with ease. This is extremely
handy while developing code. Say you're not sure if foo 1.4.0 really
requires bar v1.2.0, or if it could work with an older version (or you
just want to try it out). Then, you run:

    setup bar 1.2.0
    setup -k foo 1.4.0

and on your PYTHONPATH you'll get foo v1.4.0 and bar v1.2.0. With
virtualenv you'd need to build yet another environment, again.


Basically, the issue I see with virtualenv is that it tries to solve a
problem (and do so just for Python) that EUPS has mostly solved years
ago (and for Python, PERL, binaries, shared libraries in general, etc.),
and at a much finer level of granularity.

Virtualenv is a great idea and a very useful package for python users
who need to sandbox certain applications. Django folks just love it.
I've used it myself on other projects. I don't have anything against
virtualenv per se. But if one already has EUPS, I'm not sure it adds
much (if any) value to it. EUPS is *far* from perfect, but for this
purpose it's functionality appears to be a (strict) superset of virtualenv.

Perhaps another way to approach this is to ask what is it that we need
virtualenv for, that we can't (easily) achieve with EUPS? What is the
problem it solves for us? It's quite possible I'm missing something.

> Please, note that in order to make eups fully complient with virtualenv,
> command
> "setup python" would require an extra feature :
> Indeed activating a virtualenv-installed python require
> /path_to_virtualenv_installed_python/bin/activate script to be sourced,
> and deactivating it require
> /path_to_virtualenv_installed_python/bin/deactivate script to be runnned.
> So i think that complete virtualenv support in eups would require and
> execute() function in table file.
> Do you think an alternate solution may be used ?
> 

This would be a good feature to have, in general.

For example, I'd use it to check that the system python (or other system
libraries) are the correct version whenever a "shadow" (say)
"python-depends" module is setup-ed.

Hope this helps,
-- 
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://research.majuric.org     Phone : +1 617 744 9003

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1