On 1/2/14 9:15 , Fabrice Jammes wrote:
>>>
>> That sounds great! Is there a git repo you can point me to to take a look?
> A draft is here :
> ssh:[log in to unmask]
> 
> Nevertheless i add to modify a little bit eups source code in order it
> works,
> so could you please use it with this eups version :
> 
> https://github.com/fjammes/eups
> 
> I'll send a pull request to Robert in order he review my minor updates.
> 

Thanks, I'll try to have a look at it soon (but I'm about to fly out for
the AAS meeting on Saturday, so if I don't get to it by then it it may
be a bit longer).

>>>
>> It's a reasonable proposal (IMHO), except that one wouldn't want to
>> hardcode the python path as /usr/bin/python, but autodetect it during
>> the install of EUPS and use that.
> I completely agree with you and i've implemented this minor feature in :
> https://github.com/fjammes/eups (cf.
> https://github.com/fjammes/eups/commit/ceba62ff1b94acb8a006155dff4f4b1686bca625)
> (please note that one "-S" has been accidentally removed in bin/eups,
> but it's been corrected in a later commit)
> Would you agree with this proposal ?

The slight issue I see is that the following could mess up EUPS:

Say I have an existing install of EUPS, and I decide to distribute
updates to EUPS itself as an EUPS package. That is, when I want a user
to upgrade, I ask them to do the following:

	eups distrib install eups v1.2.3
	eups declare -c eups v1.2.3

Now, if, at the point of eups distrib install, the user has had a
version of python setup-ed that is also distributed through EUPS (say,
anaconda 1.8.0), eups would hard-code the path to that python. Then,
when the user upgrades to a different python, by, say:

	eups distrib install anaconda 1.9.0
	eups declare -c anaconda 1.9.0

and removes the old version:

	eups remove anaconda 1.8.0

their EUPS will stop working (and they won't be warned at all that their
EUPS implicitly depends on anaconda 1.8.0).

This is not a made-up example; distributing Eups through Eups is
legal and possible.

Thinking more about it, it's really the '-S' on python command line that
is the culprit here. Serge has reported that w/o it, Python succesfully
imports traceback even within the virtualenv. Robert, do you recall the
reason why -S needs to be there?

>>
>> There may be systems that don't require /usr/bin/python to be installed
>> (e.g., various *BSDs), so hardcoding the path would prevent a
>> non-privileged user from installing EUPS with their own python.
>>
>> That said:
>>
>>> Note that i met this issue while switching from system python to
>>> virtualenv python during Qserv build process (cf.
>>> http://hsca.ipmu.jp:8080/question/167/installing-virtualenv-with-eups/).
>> I think we need to understand why EUPS needs to run python with -S;
>> that's what's causing the issue with virtualenv.
>>
>> Btw., are you sure it really does this? I just grepped through the
>> source and couldn't find any invocations with '-S'.
> Yes, "-S" is used both in bin/eups and in bin/eups_setup.
> I suppose '-S" option restricts possible imports to standard library
> modules only, and make python run faster.
> Furthermore, it prevents eups developers to use non-standard modules.

OK, I see it now (I was looking at a ~year old version of EUPS, that's
why it wasn't there). Let's see what Robert says about why it's necessary.

>>
>> Could it be that it's virtualenv has a bug somwehere? It's a really
>> bizarre error, since traceback is a simple pure Python module.
> I've checked quickly virtualenv source code and it seems it only import
> a restricted subset of python standard library.
> For 2.x python version, "traceback" module isn't in the subset.

Serge reports that w/o -S, the import of traceback worked for him (see
the third comment in
http://hsca.ipmu.jp:8080/question/167/installing-virtualenv-with-eups/)

>>
....[snip]....
>>
>> Perhaps another way to approach this is to ask what is it that we need
>> virtualenv for, that we can't (easily) achieve with EUPS? What is the
>> problem it solves for us? It's quite possible I'm missing something.
> I completely agree with your explanations.
> Nevertheless, virtualenv garantee that Qserv will only use
> eups-installed python libraries :
> For example, it will prevents conflicts between libraries installed in
> /usr/lib64/python2.6/site-packages/ , and libraries installed by eups.

As EUPS products are prepended to PYTHONPATH, they'll always take
precedence, *unless* someone installed a package into system environment
with setuptools (which should _NEVER_ be done, because of the evil
PYTHONPATH manipulation setuptools does via .pth files). 

> Is it possible to do the same thing with eups only, without virtualenv?
> Indeed, I think it's an interesting feature for a service like Qserv,
> which needs stability and isolation.
> 

If one wants completely isolate a product from modules installed
system-wide, then the safest and most foolproof way is to build one's own
python. That is actually what we do currently with EUPS.

So, to achieve complete isolation, you'd do:

	setup python-v2.7
	setup ... the product you want isolated ...

and that's it. If you want that product to run with system python, just
don't type the first line. That's what I like about the EUPS solution, it's very
flexible (with virtualenv, you'd need to install it twice, I think).

> K.T. solution looks elegant and may allow to mix virtualenv and eups :/
> /
> 
> /"We already have the PYTHONPATH manipulation in the rest of the
> LSST Stack, so I think it's best for now to continue with that.  In the
> longer run, I would like to see all LSST Stack Python modules installed
> into a single directory, almost certainly an LSST-specific one and not
> the system site-packages -- but they would be swapped in and out on
> setup of the individual packages through the use of symbolic links."/
> 

I agree, but that's orthogonal to the virtualenv question.

> 
> Indeed, here
> 
> setup bar 1.2.0
> 
> wouldn't update the PYTHONPATH, but would add/remove symlinks in
> virtualenv python site-package directory. These symlinks would points to
> bar python modules)
> 

It's functionally the same -- K-T's concern (I believe) comes
more from a performance and reliability perspective.

> I've seen that your example eupspkg build scripts support system python.
> So i agree to switch to system python if you think it's, for now, the
> good way to do.
> Nevertheless, I think it would be good that the Qserv team validate this
> choice (indeed they might be good reasons to use Qserv with virtualenv
> that i may ignore).
> 

It's definitely OK to try both -- we'll learn new things along the way!

The way I'd think about it is that python in virtualenv is just another 
kind of "system python" that eups should be able to work with.

Cheers,
-- 
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://research.majuric.org     Phone : +1 617 744 9003

-- 
Mario Juric,
Sent from my Android device with K-9 Mail. Please excuse my brevity.

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1