Hi all,
On my side, I think that I'm on the same page that Andy,
I'll like to rely as more as possible on python
standard logging in python code:
- this is standard
- this is tested
- this is maintained
- this is documented
- this does not implies use of a third-party lib (i.e. log4cxx)
=> much more easy to write and maintain
I think having lsst.log as a backend to python standard logging
is a reasonable solution. Could we work on having a standard,
documented and easy way to do this?
Cheers,
Fabrice
On 03/03/2015 09:26 AM, Salnikov, Andrei A. wrote:
> Hi Jacek, K-T,
>
> I would like to argue that (almost) all logging from Python
> should be done via standard logging and not lsst.log. Actually
> I do not care much about science code for which we know that
> output will end up in log4cxx one way or another (but even for
> that code reducing dependencies on lsst-specific packages may
> be beneficial). But it's more important to me to have common
> packages like db usable in all environments. If db uses lsst.log
> it basically means that any app that uses that package (and there
> will be plenty of those) is required to redirect all logging to
> lsst.log, which I think is an overkill.
>
> The idea that packages that depend on standard python logging are
> less usable for science applications does not sound right to me.
> First there are a lot of packages that already depend on standard
> logging and we are not going to replace logging in those packages
> with lsst.log. Second, because of that, science applications (and
> some Qserv apps which mix C++ and Python) will be practically
> required to forward all python logging to lsst.log. But this needs
> to be done on per-application basis, not library basis. Redirecting
> logging output to lsst.log could probably be done via log file
> but that looks extremely ugly (two files to configure logging),
> I think it's much easier to do it in python code, it's basically
> two lines of code (and we can reduce it to one line if needed).
>
> Cheers,
> Andy
>
> Jacek Becla wrote on 2015-03-03:
>> K-T,
>>
>> I forgot to mention that the issue is only with python-side
>> of things. We already adopted lsst logging in Qserv C++ world.
>>
>>> small Python-only
>>> scripts that are not expected to produce production logs
>> I suspect Qserv does not fully fall into that category,
>> although our python scripts are not huge.
>>
>> So... if we
>> - use lsst logging in qserv-C++
>> - use python logging in qserv-python and point it to
>> lsst logging via config
>> - use lsst logging in python db module (which is called from
>> qserv, but only from qserv python admin scripts)
>> would that be reasonable?
>>
>> Hmm, I think I heard sentiment within the group that admin
>> scripts would rather use standard logging, not lsst logging.
>>
>> My feeling is that this is getting convoluted.
>>
>> Jacek
>>
>>
>>
>> On 03/03/2015 07:43 AM, Kian-Tat Lim wrote:
>>> Jacek,
>>>
>>>> We need to come back to the topic of logging. Andy (S) is telling
>>>> me that we recently decided to stick in Qserv with native python
>>>> log, and attach that log stream to the lsst logging. I do remember
>>>> we discussed this, but somehow it didn't fully occur to me that we
>>>> are not going to use our lsst logging in Qserv directly. Because...
>>>> if indeed that is the case, it does not make much sense to introduce
>>>> it in the "db" module (which I just did in DM-2137), which means
>>>> it does not make sense to use it in metaserv/imgserv/dbserv/webserv.
>>>>
>>>> So, I would like us to be completely clear about it before we
>>>> go that route. (I am not very comfortable with going that route).
>>>> And I want to make sure K-T is not going to shoot us when he
>>>> discovers our plans. :)
>>> C++ code needs to log via log4cxx. Having C++ log via the
>>> Python logger is, as far as I know, 1) difficult and 2) impossible if
>>> C++ code is not called from Python, which may still occur at times.
>>>
>>> Python code can log via log4cxx using lsst.log. It can also log
>>> via the Python library logger. The library logger can be configured to
>>> output to log4cxx, but I think that generally requires two configuration
>>> files, which I have generally resisted; I'm not sure if it also requires
>>> an actual code change to import lsst.log.
>>>
>>> The compromise I had approved before was that small Python-only
>>> scripts that are not expected to produce production logs could use the
>>> Python library logger only.
>>>
>>> But logging in packages that are shared with the Science
>>> Pipelines portion of the Stack (like db or sphgeom) needs to take more
>>> care to be compatible with C++ and its usage.
>>>
>>> The requirements I have in mind are:
>>>
>>> 1) Minimize the number of places that logging configuration is stored.
>>> Even if logging users tend to be more sophisticated developers, I think
>>> having many configuration locations makes our system hard to use.
>>>
>>> 2) Generate log outputs that are consistent in terms of format and
>>> content. Having non-ISO times (or no times at all), components in
>>> different orders, etc. makes interpretation hard.
>>>
>>> 3) Be able to configure logging at a fine grain (definitely package
>>> level; usually class level or even below).
>>>
>>> 4) Permit urgent log messages to be sent via DM Messages (currently
>>> known as ctrl_events). This will allow rapid response to problems as
>>> opposed to waiting for log files to be collected.
>>>
>>> 5) Have the primary log output from a single "application" go to a
>>> single place. I think it's a lot easier to debug if all the information
>>> is collected together and not scattered across multiple files.
>>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|