Should have sent to this list too..




-------- Original Message --------
Subject: notes from discussion about logging (db team + Steve)
Date: Tue, 08 Oct 2013 11:57:27 -0700
From: Jacek Becla <[log in to unmask]>
Organization: SLAC National Accelerator Laboratory
To: LSST Data Management <[log in to unmask]>

Steve, Serge, Daniel, Bill, Douglas, Jacek


Purpose of the meeting: to capture database
requirements and sync thoughts of the db team,
so that we are well prepared for DM-wide
discussion about logging.


action items:
  - capture requirements / statement what is needed
    a) db team is capturing db-related input through:
       https://dev.lsstcorp.org/trac/wiki/db/Qserv/Logging
       (will add a section based on this meeting discussions)
    b) need to do that for middleware / DM-wide. KT will
       surely coordinate
  - document (summarize) what we already learned about
    existing tools
  - Bill will experiment with most-promising systems,
    will migrate the logging system he wrote for Qserv,
    without committing code, just to get some hands-on
    experience. ~1 day worth of work


Requirements (brought up at the meeting):
  - want flexible system, every piece of code might
    generate different structure. So key/value best
  - want to impose some structure, like timestamp,
    thread id, component id
  - want to dynamically turn on/off parts of logging,
    and query immediately after
  - estimate expected rate... 1 million/sec aggregate???
  - want to easily filter out subsets, drill in various
    directions, eg all info for short window of time,
    all info for a given component, etc
  - want to query on local logs, not only on centralized
    loging server
  - not too concerned if we lose a line of log here and there
  - need to support simple, easy logging for developers,
    eg log to screen or file is good, setting up a server
    is bad


observation: different people have different idea
about logging levels, need project-level policy


best candidates:
  - Apache Flume (Java)
  - Kafka (Java)
  - Facebook's Scribe (C++)


Apache flum looks most attractive
  - can monitor directories, watch files
  - free format structure
  - common solution is to put data in hdfs,
    query using hive, sql-like api
  - can be setup with "routers", where router is collecting
    logs for local application and data never hits
    logging server
  - application could write to a file (or cout),
    flum would watch the files. Files can be structured,
    eg json


binary vs text logs
  - [Daniel:]
     - want to work with logs without writing text processing
       - if purely text, need to do text parsing
     - happy if we have both binary and text
     - but text only is not good enough


Steve:
  - free form definitely needed (apps team expressed this
    many times)
  - pex harness, didn't work (logging daemon writing to db).
    whole backend didn't keep up
  - bigger / larger scale requirements - need to check w/KT


Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1