Attendees: Serge, Douglas, Daniel, Jacek
Agenda:
- W2014 planning
- data loading design
- data ingest design
- merge engine
- documenting failure modes
W2014 db/qserv tasks
--------------------
- see https://dev.lsstcorp.org/trac/wiki/db/W2014
- add work on shared scans to implement comments from ticket #3002?
- no need to implement all suggestions now, most of them
are for future (part II) implementation
--> so just capture for now (link that ticket to the shared
scan doc, and/or worker redesign page)
- work on parser?
--> finish basic join syntax - code is now in Daniel's personal
branch origin/u/danielw/joinSyntax1
--> add task "research getting rid of subchunks"
--> add: "polishing query intermediate representation"
--> logging: split into software logging API", and
"distributed log collection", mention "incl basic prototyte"
Data loading
------------
- see detailed design (based on discussion Douglas+Jacek) at:
https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading
- K-T's comment "looks like M/R"
- we could leverage hadoop, but would the DM team be
happy if we relied on hadoop?
--> add to "to discuss with DM team" list
Data ingest
-----------
Next design discussion: planning to discuss data ingest
- Serge, Jacek, Douglas, (maybe others?) sometime next week
- we want to propose dividing line between pipelines and ingest
- we want to make suggestions such as "ingest = just ingest,
don't do astronomy computation, unit conversion, etc as
part of ingest, etc...
Once we have draft design of data ingest, we want to discuss
all open questions that arise from data ingest and data loading
draft design with the broader DM team
- realistically, probably after FDR
Merge engine
------------
- Run test of merge engine to check if it be used instead
of subchunks. See:
https://dev.lsstcorp.org/trac/wiki/db/tests/MySql/MergeEngine2
- conclusions: NOT good for subchunking
- might be useful to look into partitioning
- But that is a low priority task
Documenting failure modes
-------------------------
- modes of failures that we need to support. See draft at:
https://dev.lsstcorp.org/trac/wiki/db/Qserv/FaultTolerance
- overall feeling, looks "pretty good" as is
- except: don't deal with Byzantine failures, these are very
expensive to handle. We don't think any other parts
of LSST system could survive that kind of failures
- so, add a list of "failure modes that we are aware of but
we don't plan to handle", and put Byzantine failure there
Wishlists
---------
(wishlists of things each of us wants to see implemented,
fixed, cleaned etc in qserv)
- we went through all wishlists
- we will document them on trac shortly
Pushing remaining changes to master branch
------------------------------------------
- push only these that are well tested
Next meeting in a week, e.g., no special meeting on Tuesday.
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
