Print

---

[log in to unmask]" type="cite">

On 12/26/13 3:02 , Fabrice Jammes wrote:

Hi Fabrice,
	Happy holidays to you too, and see for some answers below!

Nevertheless, i still need some help on some small details, in order to
converge towards the right direction.

*1. *At the moment, the build procedure relies on HSC build style (like
it is what was recommended by Paul Price one month ago, cf.
http://hsca.ipmu.jp:8080/question/128/more-information-about-eups-create-distrib/)
It builds and installs nearly all Qserv source dependencies (but not
scisql and Qserv itself) on both Ubuntu 13.04 and SL6.2.
For information, i did not use nor lssteups or sconsutils (i only have
the qserv package to build with scons, and Qserv only links with
protobuf, xrootd and mysql).

That sounds great! Is there a git repo you can point me to to take a look?

[log in to unmask]" type="cite">

- Do you confirm i have to move all these build script to pkgbuild right
now ? Is pkgbuild stable enough ? If yes, does
https://github.com/RobertLuptonTheGood/eups.git master branch support
pkgbuild, or shall i use
an other branch ?

I just got EupsPkg (formerly known as pkgbuild) into a state where it's
ready for review. It's right now at 'eupspkg' branch of my EUPS fork in
github:

	https://github.com/mjuric/eups/tree/eupspkg

I plan to make a pull request to Robert (and ask others for comments as
well) imminently, and barring huge architectural disagreements I think
it will get merged into EUPS proper soon (perhaps ~two weeks or so).

Given that, I think it makes sense to start transitioning your build
scripts to eupspkg. It hope it won't be hard -- I've written
documentation in the docstring of eupspkg module:

	https://github.com/mjuric/eups/blob/eupspkg/python/eups/distrib/eupspkg.py

, made examples for all LSST stack external dependencies:

	https://dev.lsstcorp.org/cgit/personal/mjuric/eupspkg/external/

, and for LSST code that relies on scons (via sconsUtils), eupspkg knows
what to do out of the box (i.e., there's no need to modify it at all or
supply a ./ups/eupspkg file).

If you point me to your existing builder scripts, I can help with
transforming them to eupspkg variants. My guess, based on doing that for
the rest of DM stack dependencies, is that it shouldn't take more than
~2-3 hours.

[log in to unmask]" type="cite">

- Do you think Qserv build should relies on sconsutils ?

I'm not familiar with the details, so I'll let K-T et al answer.

- Do you thinks sconsutils may be compatible with rpm format ?

I think it should be possible to write a fully automated eupspkg-to-spec
converter. All information needed to obtain and build the source is in
the eupspkg scripts.

- Do you confirm Qserv build musn't rely on lssteups (I didn't notice
the use of this tool while i investigated HSC build scripts.) ?

The plan is for lssteups to be superseded by eupspkg; therefore, it
shouldn't be used any more.

More comments below:

For the next three points, i answer you between the lines below :

On 12/20/2013 07:55 PM, Kian-Tat Lim wrote:

Fabrice,

Maybe it would be better if eups always run with system python ?

	In the not-too-distant future (maybe even in January or
February), I think we want to convert the LSST Stack to run with the
system Python in all cases, including eups.

Indeed for now it switch to eups-installed python once command
"setup python" is performed.

	"setup python" would go away.  But I'm not sure what the problem
is with switching Pythons.

*2. *The problem is met with next hypothesis :
- eups is compatible with python version i, but not with python version
j, and eups is used to install a software, named SOFT, which require
python version j. (note that j can even be lower than i)
During SOFT install, eups will install python version j, and in current
eups version, it will itself switch to python version j, and then the
whole install procedure may break. Indeed all call to eups may fails
because of the use of python version j,
so it may even be not possible to go back (unsetup command may fails).

That's why i propose that, during eups installation, all call to eups
executables are forced to be done with python version i. Note that all
other calls to python done by the build scripts will use environment
python, i.e. python version j.

(For exemple, to use this solution, eups/bin/eups would be simply
updated like during by eups install process :
--- a/bin/eups
+++ b/bin/eups
@@ -3,4 +3,4 @@
 #  #!/usr/bin/env python options
 #
 unset PYTHONSTARTUP
-*python* -S $EUPS_DIR/bin/eups_impl.py "$@"
+*/usr/bin/python* -S $EUPS_DIR/bin/eups_impl.py "$@"
)

This solution, trivial to implement, would make eups more stable and
more generic, don't you think so ?
(I expose briefly this issue to Christian Arnault, french manager for
LSST computing activities and author of CMT, the atlas build system, and
he agreed with me.)

It's a reasonable proposal (IMHO), except that one wouldn't want to
hardcode the python path as /usr/bin/python, but autodetect it during
the install of EUPS and use that.

[log in to unmask]" type="cite">

There may be systems that don't require /usr/bin/python to be installed
(e.g., various *BSDs), so hardcoding the path would prevent a
non-privileged user from installing EUPS with their own python.

That said:

Note that i met this issue while switching from system python to
virtualenv python during Qserv build process (cf.
http://hsca.ipmu.jp:8080/question/167/installing-virtualenv-with-eups/).

I think we need to understand why EUPS needs to run python with -S;
that's what's causing the issue with virtualenv.

Btw., are you sure it really does this? I just grepped through the
source and couldn't find any invocations with '-S'.

[log in to unmask]" type="cite">

Could it be that it's virtualenv has a bug somwehere? It's a really
bizarre error, since traceback is a simple pure Python module.

[log in to unmask]" type="cite">

More below:

[log in to unmask]" type="cite">

	Regarding its use with virtualenv, EUPS has been designed with
virtual-env like functionality in mind -- specifically, it manages the
PYTHON_PATH for you. So using the two together is likely to generate
conflicts (though it may not be impossible).

One of the main advantage of using virtualenv is that install of
python libraries via eups is completely transparent : you don't have
to manage a dedicated PYTHONPATH in eups build scripts of python
libraries.
It also isolate eups-installed python and then prevent conflicts
between system python user libraries and eups-installed python user
libraries.
Of course, if you prefer, i can set up build script to install each
Qserv python lib in a dedicated directory and then add it to
PYTHONPATH.

	We already have the PYTHONPATH manipulation in the rest of the
LSST Stack, so I think it's best for now to continue with that.  In the
longer run, I would like to see all LSST Stack Python modules installed
into a single directory, almost certainly an LSST-specific one and not
the system site-packages -- but they would be swapped in and out on
setup of the individual packages through the use of symbolic links.

*3. *This long-term solution looks fine. So i confirm i will implement
PYTHONPATH manipulation in my next version of Qserv packaging tool.

	One of the main reasons to use the system Python is to allow the
user to incorporate other packages into their usage of the LSST Stack.
virtualenv does not work well with this usage model.  This may not be an
issue with qserv, however, which is server-side and can insist on a more
rigorously-controlled environment.

*
4. *Qserv now relies on virtualenv but it also may works with system-python.

An interesting feature of virtualenv in a rigorously-controlled
environment is that it may switch PYTHONPATH while its
activated/deactivated.

But that is *exactly* the problem that EUPS tries to solve, not just for
Python modules but for other code as well, and with much more granularity.

Think of EUPS as being an "uber-virtualenv", where you can choose to
activate or deactivate not just the entire environment at once, but each
*individual package*. The equivalent of 'activate' with EUPS is 'setup',
and 'deactivate' is 'unsetup'.

For example, say I have a Python code foo v1.2.3 that depends on a
Python code bar v1.0.0, a shared library named baz v1.1.1 (i.e.,
libbaz.so.1.1.1), and a command-line utility named qux 3.2.1:

    foo v1.2.3
    +-- bar v1.0.0
    +-- baz v1.1.1
    +-- qux v3.2.1

Say they're all installed in an EUPS stack. I can set them all up by
running:

    setup foo 1.2.3

This will bring foo and bar onto PYTHONPATH, baz onto LD_LIBRARY_PATH,
and qux onto PATH. Virtualenv's 'activate' would just do the first of
these (at least that was the case when I last used it). To do the
reverse, I'd run 'unsetup foo', and both PYTHONPATH and LD_LIBRARY_PATH
would both get properly cleaned up. That is what virtualenv's
'deactivate' does, but, again, just for Python modules.

Now, if I wanted to install a new version of foo (say v1.3.0), I can
easily do that with EUPS without overwriting the existing one. It won't
interfere with v1.2.3, and it will reuse the already installed bar and
baz (assuming it still depends on them). With virtualenv, I'd need to
build a whole new environment just to update this one package as v1.3.0
will want to overwrite the version already in the virtualenv container.
This is an issue when there may be 50+ other packages in the stack (and
with LSST, there are).

Similarly, if I next installed foo v1.4.0 that now requires bar v1.2.0,
I could do that easily with EUPS; neither will interfere with other
versions, and when I run:

    setup foo 1.4.0

EUPS will know it needs to also setup bar 1.2.0 (because it tracks
dependencies). Again, with virtualenv, I'd need to build a whole new
environment; it has no notion of internal product dependencies.

Finally, EUPS lets you "mix and match" with ease. This is extremely
handy while developing code. Say you're not sure if foo 1.4.0 really
requires bar v1.2.0, or if it could work with an older version (or you
just want to try it out). Then, you run:

    setup bar 1.2.0
    setup -k foo 1.4.0

and on your PYTHONPATH you'll get foo v1.4.0 and bar v1.2.0. With
virtualenv you'd need to build yet another environment, again.


Basically, the issue I see with virtualenv is that it tries to solve a
problem (and do so just for Python) that EUPS has mostly solved years
ago (and for Python, PERL, binaries, shared libraries in general, etc.),
and at a much finer level of granularity.

Virtualenv is a great idea and a very useful package for python users
who need to sandbox certain applications. Django folks just love it.
I've used it myself on other projects. I don't have anything against
virtualenv per se. But if one already has EUPS, I'm not sure it adds
much (if any) value to it. EUPS is *far* from perfect, but for this
purpose it's functionality appears to be a (strict) superset of virtualenv.

Perhaps another way to approach this is to ask what is it that we need
virtualenv for, that we can't (easily) achieve with EUPS? What is the
problem it solves for us? It's quite possible I'm missing something.

"We already have the PYTHONPATH manipulation in the rest of the
LSST Stack, so I think it's best for now to continue with that.  In the
longer run, I would like to see all LSST Stack Python modules installed
into a single directory, almost certainly an LSST-specific one and not
the system site-packages -- but they would be swapped in and out on
setup of the individual packages through the use of symbolic links."

setup bar 1.2.0

[log in to unmask]" type="cite">

Please, note that in order to make eups fully complient with virtualenv,
command
"setup python" would require an extra feature :
Indeed activating a virtualenv-installed python require
/path_to_virtualenv_installed_python/bin/activate script to be sourced,
and deactivating it require
/path_to_virtualenv_installed_python/bin/deactivate script to be runnned.
So i think that complete virtualenv support in eups would require and
execute() function in table file.
Do you think an alternate solution may be used ?

This would be a good feature to have, in general.

For example, I'd use it to check that the system python (or other system
libraries) are the correct version whenever a "shadow" (say)
"python-depends" module is setup-ed.

Hope this helps,

Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1