Attendees: Robert Lupton, K-T Lim, Bill Chickering,
            Mike Freemon, Serge Monkewitz, Jacek


High level summary
==================
  - application-level logging:
     - considering using log4cxx, plus one-line interface
       and configuration
     - swig it to python
  - distributed logging:
     - considering using Apache Flume
  - next steps:
    - build a prototype
    - evaluate
    - make final decision (this is probably SAT level)
    - expecting to have app-level prototype in late January,
      with Flume in February [Bill will build it]


Related reading
===============
https://dev.lsstcorp.org/trac/wiki/db/Qserv/Logging
(it will be updated shortly after these notes are sent out)


Detailed notes
==============

the wiki page does not cover our existing event system
  - it could be considered as an option for distributed logging
  - although it has many issues


google logging
  - automatically selects module name (we dislike it)
  - doesn't seem very popular


yes, pex_logging has optimizations and tries to minimize
computation if logging turned off


pay attention to performance, e.g., ideally Robert would
like an option to compile out logging all together


it'd be useful to define even more levels than log4cxx offers
(e.g., several levels within "debug") - need to research if
that is doable


Robert dislikes the api in pex_logging, want something
as easy as printf, a.la.:

    LOG("a.b.c", n, message, args, ...);

   where "a.b.c" is the hierarchical component name, n is
   the severity, and args are printf/boost::format/stream
   style arguments for the message.


need to be able to attach metadata to individual log messages


log4cxx directly vs wrapping?
  - log4cxx plus one-line simple format, and dealing with
    configuration


need to be able to dynamically turn on/off distributed logging
(isolate our software from distributed logging)
  - proposed model: configure system to produce log in log4j
    format, distributed logging simply consumes the data


same logging system from python and C++ or they can diverge?
  - same. It is a requirement!


logging from python
  - swig the c++ implementation
  - yes, even for simple, python-only admin tools


configuring logging
  - from c++: through api or a text file
  - from python: through a text file only


two main uses for logging:
  - manual debugging
  - automated parsing of logs
    - want key/value for that
       - can configure log4cxx to write in json format


problem with existing event system
  - ingesting into rdbms slow, querying key/value in rdbms slow
  - it does not batch
  - issues with multi-threading
  - but using event system as a transport protocol is an option,
    it is fast, scales, reliable


sending log data to distributed logging system
  - through local files, or streaming directly
  - if local files: space mgmt issues (disks getting full)
    - but acts as a buffer, if distrib logging down, can resync
      and catch up


Flume vs Kafka
  - Flum is simpler, easier to deploy than Kafka
  - Kafka focuses a lot on high throughput / high performance
  - both use ZooKeeper
  - Flume is not just for hdfs
  - both solid, well supported, Apache products
  --> prototype with Flume


if we use Flume, would we still use event system for monitoring?
Options:
  - use event system
  - do event monitoring in Flume
  - generate events from Flume and use event system after Flume


This discussion is all about application level logging, not for
system level logging (done by NCSA)
  - NCSA has good tools that they use and like
  - Mike will send some info, if it looks interesting to us
    we will schedule a phone call to discuss


Jacek




On 12/17/2013 10:53 PM, Jacek Becla wrote:
> Hi,
>
> I've put together a sketch proposal for logging system
> based on earlier research done by our student Bill
> on distributed logging systems, plus a couple of
> hours of research/reading up about pex_logging,
> log4cxx, Flume and Kafka, see:
>
> https://dev.lsstcorp.org/trac/wiki/db/Qserv/Logging
>
> I'm planning to discuss it tomorrow (Wed) at 11:00
> pacific with Bill and K-T.
>
> If anyone is interested in joining, we'll keep
> the line open: 866 740 1260, pass 9268664.
>
> Jacek
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1