Hi Jacek, K-T,

I would like to argue that (almost) all logging from Python 
should be done via standard logging and not lsst.log. Actually
I do not care much about science code for which we know that
output will end up in log4cxx one way or another (but even for 
that code reducing dependencies on lsst-specific packages may 
be beneficial). But it's more important to me to have common
packages like db usable in all environments. If db uses lsst.log 
it basically means that any app that uses that package (and there
will be plenty of those) is required to redirect all logging to 
lsst.log, which I think is an overkill.

The idea that packages that depend on standard python logging are
less usable for science applications does not sound right to me.
First there are a lot of packages that already depend on standard 
logging and we are not going to replace logging in those packages 
with lsst.log. Second, because of that, science applications (and
some Qserv apps which mix C++ and Python) will be practically 
required to forward all python logging to lsst.log. But this needs 
to be done on per-application basis, not library basis. Redirecting
logging output to lsst.log could probably be done via log file
but that looks extremely ugly (two files to configure logging),
I think it's much easier to do it in python code, it's basically 
two lines of code (and we can reduce it to one line if needed).

Cheers,
Andy

Jacek Becla wrote on 2015-03-03:
> K-T,
> 
> I forgot to mention that the issue is only with python-side
> of things. We already adopted lsst logging in Qserv C++ world.
> 
>> small Python-only
>> scripts that are not expected to produce production logs
> 
> I suspect Qserv does not fully fall into that category,
> although our python scripts are not huge.
> 
> So... if we
>   - use lsst logging in qserv-C++
>   - use python logging in qserv-python and point it to
>     lsst logging via config
>   - use lsst logging in python db module (which is called from
>     qserv, but only from qserv python admin scripts)
> would that be reasonable?
> 
> Hmm, I think I heard sentiment within the group that admin
> scripts would rather use standard logging, not lsst logging.
> 
> My feeling is that this is getting convoluted.
> 
> Jacek
> 
> 
> 
> On 03/03/2015 07:43 AM, Kian-Tat Lim wrote:
>> Jacek,
>> 
>>> We need to come back to the topic of logging. Andy (S) is telling
>>> me that we recently decided to stick in Qserv with native python
>>> log, and attach that log stream to the lsst logging. I do remember
>>> we discussed this, but somehow it didn't fully occur to me that we
>>> are not going to use our lsst logging in Qserv directly. Because...
>>> if indeed that is the case, it does not make much sense to introduce
>>> it in the "db" module (which I just did in DM-2137), which means
>>> it does not make sense to use it in metaserv/imgserv/dbserv/webserv.
>>> 
>>> So, I would like us to be completely clear about it before we
>>> go that route. (I am not very comfortable with going that route).
>>> And I want to make sure K-T is not going to shoot us when he
>>> discovers our plans. :)
>> 
>> 	C++ code needs to log via log4cxx.  Having C++ log via the
>> Python logger is, as far as I know, 1) difficult and 2) impossible if
>> C++ code is not called from Python, which may still occur at times.
>> 
>> 	Python code can log via log4cxx using lsst.log.  It can also log
>> via the Python library logger.  The library logger can be configured to
>> output to log4cxx, but I think that generally requires two configuration
>> files, which I have generally resisted; I'm not sure if it also requires
>> an actual code change to import lsst.log.
>> 
>> 	The compromise I had approved before was that small Python-only
>> scripts that are not expected to produce production logs could use the
>> Python library logger only.
>> 
>> 	But logging in packages that are shared with the Science
>> Pipelines portion of the Stack (like db or sphgeom) needs to take more
>> care to be compatible with C++ and its usage.
>> 
>> 	The requirements I have in mind are:
>> 
>> 1) Minimize the number of places that logging configuration is stored.
>> Even if logging users tend to be more sophisticated developers, I think
>> having many configuration locations makes our system hard to use.
>> 
>> 2) Generate log outputs that are consistent in terms of format and
>> content.  Having non-ISO times (or no times at all), components in
>> different orders, etc. makes interpretation hard.
>> 
>> 3) Be able to configure logging at a fine grain (definitely package
>> level; usually class level or even below).
>> 
>> 4) Permit urgent log messages to be sent via DM Messages (currently
>> known as ctrl_events).  This will allow rapid response to problems as
>> opposed to waiting for log files to be collected.
>> 
>> 5) Have the primary log output from a single "application" go to a
>> single place.  I think it's a lot easier to debug if all the information
>> is collected together and not scattered across multiple files.
>> 
> 
> ########################################################################
> Use REPLY-ALL to reply to list
> 
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1