Print

---

Hello Mario,

All the best and a happy new year 2014, and thanks for your usefull answer,
i added some answers, comments and questions below.

On 12/31/2013 01:16 PM, Mario Juric wrote:
> On 12/26/13 3:02 , Fabrice Jammes wrote:
> Hi Fabrice,
> 	Happy holidays to you too, and see for some answers below!
>
>> Nevertheless, i still need some help on some small details, in order to
>> converge towards the right direction.
>>
>> *1. *At the moment, the build procedure relies on HSC build style (like
>> it is what was recommended by Paul Price one month ago, cf.
>> http://hsca.ipmu.jp:8080/question/128/more-information-about-eups-create-distrib/)
>> It builds and installs nearly all Qserv source dependencies (but not
>> scisql and Qserv itself) on both Ubuntu 13.04 and SL6.2.
>> For information, i did not use nor lssteups or sconsutils (i only have
>> the qserv package to build with scons, and Qserv only links with
>> protobuf, xrootd and mysql).
>>
> That sounds great! Is there a git repo you can point me to to take a look?
A draft is here :
ssh:[log in to unmask]

Nevertheless i add to modify a little bit eups source code in order it 
works,
so could you please use it with this eups version :

https://github.com/fjammes/eups

I'll send a pull request to Robert in order he review my minor updates.

>
>> - Do you confirm i have to move all these build script to pkgbuild right
>> now ? Is pkgbuild stable enough ? If yes, does
>> https://github.com/RobertLuptonTheGood/eups.git master branch support
>> pkgbuild, or shall i use
>> an other branch ?
> I just got EupsPkg (formerly known as pkgbuild) into a state where it's
> ready for review. It's right now at 'eupspkg' branch of my EUPS fork in
> github:
>
> 	https://github.com/mjuric/eups/tree/eupspkg
>
> I plan to make a pull request to Robert (and ask others for comments as
> well) imminently, and barring huge architectural disagreements I think
> it will get merged into EUPS proper soon (perhaps ~two weeks or so).
>
> Given that, I think it makes sense to start transitioning your build
> scripts to eupspkg. It hope it won't be hard -- I've written
> documentation in the docstring of eupspkg module:
>
> 	https://github.com/mjuric/eups/blob/eupspkg/python/eups/distrib/eupspkg.py
>
> , made examples for all LSST stack external dependencies:
>
> 	https://dev.lsstcorp.org/cgit/personal/mjuric/eupspkg/external/
>
> , and for LSST code that relies on scons (via sconsUtils), eupspkg knows
> what to do out of the box (i.e., there's no need to modify it at all or
> supply a ./ups/eupspkg file).
>
> If you point me to your existing builder scripts, I can help with
> transforming them to eupspkg variants. My guess, based on doing that for
> the rest of DM stack dependencies, is that it shouldn't take more than
> ~2-3 hours.
Thanks for your help. I quickly read some of your eupspkg files and they 
look very simple, nevertheless i would be pleased if you could help me
with transforming qserv build scripts to eupspkg variants.

More comments below :

>
>> - Do you think Qserv build should relies on sconsutils ?
> I'm not familiar with the details, so I'll let K-T et al answer.
>
>> - Do you thinks sconsutils may be compatible with rpm format ?
> I think it should be possible to write a fully automated eupspkg-to-spec
> converter. All information needed to obtain and build the source is in
> the eupspkg scripts.
>
>> - Do you confirm Qserv build musn't rely on lssteups (I didn't notice
>> the use of this tool while i investigated HSC build scripts.) ?
>>
> The plan is for lssteups to be superseded by eupspkg; therefore, it
> shouldn't be used any more.
>
> More comments below:
>> For the next three points, i answer you between the lines below :
>>
>> On 12/20/2013 07:55 PM, Kian-Tat Lim wrote:
>>> Fabrice,
>>>
>>>> Maybe it would be better if eups always run with system python ?
>>> 	In the not-too-distant future (maybe even in January or
>>> February), I think we want to convert the LSST Stack to run with the
>>> system Python in all cases, including eups.
>>>
>>>> Indeed for now it switch to eups-installed python once command
>>>> "setup python" is performed.
>>> 	"setup python" would go away.  But I'm not sure what the problem
>>> is with switching Pythons.
>> *2. *The problem is met with next hypothesis :
>> - eups is compatible with python version i, but not with python version
>> j, and eups is used to install a software, named SOFT, which require
>> python version j. (note that j can even be lower than i)
>> During SOFT install, eups will install python version j, and in current
>> eups version, it will itself switch to python version j, and then the
>> whole install procedure may break. Indeed all call to eups may fails
>> because of the use of python version j,
>> so it may even be not possible to go back (unsetup command may fails).
>>
>> That's why i propose that, during eups installation, all call to eups
>> executables are forced to be done with python version i. Note that all
>> other calls to python done by the build scripts will use environment
>> python, i.e. python version j.
>>
>> (For exemple, to use this solution, eups/bin/eups would be simply
>> updated like during by eups install process :
>> --- a/bin/eups
>> +++ b/bin/eups
>> @@ -3,4 +3,4 @@
>>   #  #!/usr/bin/env python options
>>   #
>>   unset PYTHONSTARTUP
>> -*python* -S $EUPS_DIR/bin/eups_impl.py "$@"
>> +*/usr/bin/python* -S $EUPS_DIR/bin/eups_impl.py "$@"
>> )
>>
>> This solution, trivial to implement, would make eups more stable and
>> more generic, don't you think so ?
>> (I expose briefly this issue to Christian Arnault, french manager for
>> LSST computing activities and author of CMT, the atlas build system, and
>> he agreed with me.)
>>
> It's a reasonable proposal (IMHO), except that one wouldn't want to
> hardcode the python path as /usr/bin/python, but autodetect it during
> the install of EUPS and use that.
I completely agree with you and i've implemented this minor feature in :
https://github.com/fjammes/eups (cf. 
https://github.com/fjammes/eups/commit/ceba62ff1b94acb8a006155dff4f4b1686bca625)
(please note that one "-S" has been accidentally removed in bin/eups, 
but it's been corrected in a later commit)
Would you agree with this proposal ?
>
> There may be systems that don't require /usr/bin/python to be installed
> (e.g., various *BSDs), so hardcoding the path would prevent a
> non-privileged user from installing EUPS with their own python.
>
> That said:
>
>> Note that i met this issue while switching from system python to
>> virtualenv python during Qserv build process (cf.
>> http://hsca.ipmu.jp:8080/question/167/installing-virtualenv-with-eups/).
> I think we need to understand why EUPS needs to run python with -S;
> that's what's causing the issue with virtualenv.
>
> Btw., are you sure it really does this? I just grepped through the
> source and couldn't find any invocations with '-S'.
Yes, "-S" is used both in bin/eups and in bin/eups_setup.
I suppose '-S" option restricts possible imports to standard library 
modules only, and make python run faster.
Furthermore, it prevents eups developers to use non-standard modules.
>
> Could it be that it's virtualenv has a bug somwehere? It's a really
> bizarre error, since traceback is a simple pure Python module.
I've checked quickly virtualenv source code and it seems it only import 
a restricted subset of python standard library.
For 2.x python version, "traceback" module isn't in the subset.
>
> More below:
More below ;-)
>>>>> 	Regarding its use with virtualenv, EUPS has been designed with
>>>>> virtual-env like functionality in mind -- specifically, it manages the
>>>>> PYTHON_PATH for you. So using the two together is likely to generate
>>>>> conflicts (though it may not be impossible).
>>>> One of the main advantage of using virtualenv is that install of
>>>> python libraries via eups is completely transparent : you don't have
>>>> to manage a dedicated PYTHONPATH in eups build scripts of python
>>>> libraries.
>>>> It also isolate eups-installed python and then prevent conflicts
>>>> between system python user libraries and eups-installed python user
>>>> libraries.
>>>> Of course, if you prefer, i can set up build script to install each
>>>> Qserv python lib in a dedicated directory and then add it to
>>>> PYTHONPATH.
>>> 	We already have the PYTHONPATH manipulation in the rest of the
>>> LSST Stack, so I think it's best for now to continue with that.  In the
>>> longer run, I would like to see all LSST Stack Python modules installed
>>> into a single directory, almost certainly an LSST-specific one and not
>>> the system site-packages -- but they would be swapped in and out on
>>> setup of the individual packages through the use of symbolic links.
>> *3. *This long-term solution looks fine. So i confirm i will implement
>> PYTHONPATH manipulation in my next version of Qserv packaging tool.
>>> 	One of the main reasons to use the system Python is to allow the
>>> user to incorporate other packages into their usage of the LSST Stack.
>>> virtualenv does not work well with this usage model.  This may not be an
>>> issue with qserv, however, which is server-side and can insist on a more
>>> rigorously-controlled environment.
>>>
>> *
>> 4. *Qserv now relies on virtualenv but it also may works with system-python.
>>
>> An interesting feature of virtualenv in a rigorously-controlled
>> environment is that it may switch PYTHONPATH while its
>> activated/deactivated.
>>
> But that is *exactly* the problem that EUPS tries to solve, not just for
> Python modules but for other code as well, and with much more granularity.
>
> Think of EUPS as being an "uber-virtualenv", where you can choose to
> activate or deactivate not just the entire environment at once, but each
> *individual package*. The equivalent of 'activate' with EUPS is 'setup',
> and 'deactivate' is 'unsetup'.
>
> For example, say I have a Python code foo v1.2.3 that depends on a
> Python code bar v1.0.0, a shared library named baz v1.1.1 (i.e.,
> libbaz.so.1.1.1), and a command-line utility named qux 3.2.1:
>
>      foo v1.2.3
>      +-- bar v1.0.0
>      +-- baz v1.1.1
>      +-- qux v3.2.1
>
> Say they're all installed in an EUPS stack. I can set them all up by
> running:
>
>      setup foo 1.2.3
>
> This will bring foo and bar onto PYTHONPATH, baz onto LD_LIBRARY_PATH,
> and qux onto PATH. Virtualenv's 'activate' would just do the first of
> these (at least that was the case when I last used it). To do the
> reverse, I'd run 'unsetup foo', and both PYTHONPATH and LD_LIBRARY_PATH
> would both get properly cleaned up. That is what virtualenv's
> 'deactivate' does, but, again, just for Python modules.
>
> Now, if I wanted to install a new version of foo (say v1.3.0), I can
> easily do that with EUPS without overwriting the existing one. It won't
> interfere with v1.2.3, and it will reuse the already installed bar and
> baz (assuming it still depends on them). With virtualenv, I'd need to
> build a whole new environment just to update this one package as v1.3.0
> will want to overwrite the version already in the virtualenv container.
> This is an issue when there may be 50+ other packages in the stack (and
> with LSST, there are).
>
> Similarly, if I next installed foo v1.4.0 that now requires bar v1.2.0,
> I could do that easily with EUPS; neither will interfere with other
> versions, and when I run:
>
>      setup foo 1.4.0
>
> EUPS will know it needs to also setup bar 1.2.0 (because it tracks
> dependencies). Again, with virtualenv, I'd need to build a whole new
> environment; it has no notion of internal product dependencies.
>
> Finally, EUPS lets you "mix and match" with ease. This is extremely
> handy while developing code. Say you're not sure if foo 1.4.0 really
> requires bar v1.2.0, or if it could work with an older version (or you
> just want to try it out). Then, you run:
>
>      setup bar 1.2.0
>      setup -k foo 1.4.0
>
> and on your PYTHONPATH you'll get foo v1.4.0 and bar v1.2.0. With
> virtualenv you'd need to build yet another environment, again.
>
>
> Basically, the issue I see with virtualenv is that it tries to solve a
> problem (and do so just for Python) that EUPS has mostly solved years
> ago (and for Python, PERL, binaries, shared libraries in general, etc.),
> and at a much finer level of granularity.
>
> Virtualenv is a great idea and a very useful package for python users
> who need to sandbox certain applications. Django folks just love it.
> I've used it myself on other projects. I don't have anything against
> virtualenv per se. But if one already has EUPS, I'm not sure it adds
> much (if any) value to it. EUPS is *far* from perfect, but for this
> purpose it's functionality appears to be a (strict) superset of virtualenv.
>
> Perhaps another way to approach this is to ask what is it that we need
> virtualenv for, that we can't (easily) achieve with EUPS? What is the
> problem it solves for us? It's quite possible I'm missing something.
I completely agree with your explanations.
Nevertheless, virtualenv garantee that Qserv will only use 
eups-installed python libraries :
For example, it will prevents conflicts between libraries installed in 
/usr/lib64/python2.6/site-packages/ , and libraries installed by eups.
Is it possible to do the same thing with eups only, without virtualenv? 
Indeed, I think it's an interesting feature for a service like Qserv, 
which needs stability and isolation.

K.T. solution looks elegant and may allow to mix virtualenv and eups :/
/

/"We already have the PYTHONPATH manipulation in the rest of the
LSST Stack, so I think it's best for now to continue with that.  In the
longer run, I would like to see all LSST Stack Python modules installed
into a single directory, almost certainly an LSST-specific one and not
the system site-packages -- but they would be swapped in and out on
setup of the individual packages through the use of symbolic links."/


Indeed, here

setup bar 1.2.0

wouldn't update the PYTHONPATH, but would add/remove symlinks in 
virtualenv python site-package directory. These symlinks would points to 
bar python modules)

I've seen that your example eupspkg build scripts support system python.
So i agree to switch to system python if you think it's, for now, the 
good way to do.
Nevertheless, I think it would be good that the Qserv team validate this 
choice (indeed they might be good reasons to use Qserv with virtualenv 
that i may ignore).

>
>> Please, note that in order to make eups fully complient with virtualenv,
>> command
>> "setup python" would require an extra feature :
>> Indeed activating a virtualenv-installed python require
>> /path_to_virtualenv_installed_python/bin/activate script to be sourced,
>> and deactivating it require
>> /path_to_virtualenv_installed_python/bin/deactivate script to be runnned.
>> So i think that complete virtualenv support in eups would require and
>> execute() function in table file.
>> Do you think an alternate solution may be used ?
>>
> This would be a good feature to have, in general.
>
> For example, I'd use it to check that the system python (or other system
> libraries) are the correct version whenever a "shadow" (say)
> "python-depends" module is setup-ed.
>
> Hope this helps,
Yes it's help a lot, thank !

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1