Attendees: Serge, Douglas, Daniel, Jacek


Agenda:
  - W2014 planning
  - data loading design
  - data ingest design
  - merge engine
  - documenting failure modes



W2014 db/qserv tasks
--------------------
  - see https://dev.lsstcorp.org/trac/wiki/db/W2014
  - add work on shared scans to implement comments from ticket #3002?
     - no need to implement all suggestions now, most of them
       are for future (part II) implementation
      --> so just capture for now (link that ticket to the shared
        scan doc, and/or worker redesign page)
  - work on parser?
    --> finish basic join syntax - code is now in Daniel's personal
      branch origin/u/danielw/joinSyntax1
  --> add task "research getting rid of subchunks"
  --> add: "polishing query intermediate representation"
  --> logging: split into software logging API", and
     "distributed log collection", mention "incl basic prototyte"


Data loading
------------
  - see detailed design (based on discussion Douglas+Jacek) at:
    https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading
  - K-T's comment "looks like M/R"
    - we could leverage hadoop, but would the DM team be
      happy if we relied on hadoop?
      --> add to "to discuss with DM team" list


Data ingest
-----------
Next design discussion: planning to discuss data ingest
  - Serge, Jacek, Douglas, (maybe others?) sometime next week
  - we want to propose dividing line between pipelines and ingest
  - we want to make suggestions such as "ingest = just ingest,
    don't do astronomy computation, unit conversion, etc as
    part of ingest, etc...

Once we have draft design of data ingest, we want to discuss
all open questions that arise from data ingest and data loading
draft design with the broader DM team
  - realistically, probably after FDR


Merge engine
------------
  - Run test of merge engine to check if it be used instead
    of subchunks. See:
    https://dev.lsstcorp.org/trac/wiki/db/tests/MySql/MergeEngine2
  - conclusions: NOT good for subchunking
  - might be useful to look into partitioning
    -  But that is a low priority task


Documenting failure modes
-------------------------
  - modes of failures that we need to support. See draft at:
    https://dev.lsstcorp.org/trac/wiki/db/Qserv/FaultTolerance
  - overall feeling, looks "pretty good" as is
  - except: don't deal with Byzantine failures, these are very
    expensive to handle. We don't think any other parts
    of LSST system could survive that kind of failures
    - so, add a list of "failure modes that we are aware of but
      we don't plan to handle", and put Byzantine failure there


Wishlists
---------
(wishlists of things each of us wants to see implemented,
fixed, cleaned etc in qserv)
  - we went through all wishlists
  - we will document them on trac shortly


Pushing remaining changes to master branch
------------------------------------------
  - push only these that are well tested


Next meeting in a week, e.g., no special meeting on Tuesday.

Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1