Attendees:
Daniel [DW]
Serge [SMM]
Douglas [DS]
Fabrice [FJ]
Jacek [JB]
Agenda:
- quick round-table status report & next steps
- W14 db tasks
- EXTASCID - debrief from mtg, next steps
- hackathon planning
- stand-up meetings
- documentation
Status report
=============
- data ingest and loading design discussions
[JB, SMM, KT, DW] and documentation [JB]
- W14 planning [JB]
- meeting with Florin Rusu (more below) [JB, DW, KT]
- code cleanup [DS]
- db trac page cleanup [JB]
- playing with Zookeeper [DS]
- work on Qserv modularization [DW, BC]
Next steps (this covers all action items identified at
the meeting, some sprinkled below throughout the notes)
=======================================================
code cleanup (want it done by next Thu)
- finish #3007 [DS]
- finish #3004, #3013 [FJ + ticket reviewers]
- when the 3 above done, will be ready for integration
with buildbot
- want all changes pushed to master by next Thu
- review Douglas' ticket [DW]
- merge 3 tickets [BC]
- concurrency fixes - put in a ticket, by early
next week [SMM]
- create new tag for Qserv that captures all
useful code lingering around, by next Thu [DS]
prep for hackathon (all items in this section are
high priority)
- document what is now done in python [JB]
- hackathon planning [JB + input from all]
- user friendly communication and syntax parsing:
Daniel has some ideas, document on trac [DW]
- advance work on modularization and present next
Thu at qserv meeting [DW, BC]
- lay out list of things that can be done
on either worker or master [DW]
- bring Mario/Robert if possible (remotely) for
discussion about building/packaging, during
hackathon week [JB]
- add to trac hackathon planning: "tentative list
of outcomes" [JB]
- take apart wishlists and document each point
raised in appropriate place [JB]
coding (low priority)
- work on objectId index [DW]
- join syntax improvements [DW]
- small tweaks to shared scan ticket [DW]
- later, after next week, when we have the new
Qserv tag: cleanup all Qserv tickets [JB?]
- follow up with Florin Rusu, check his code [DW]
unclassified
- talk to Robyn about integrating with buildbot
sometime later next week [JB]
- send Fabrice info about policies/restrictions
regarding in2p3 contributions *during
construction* [JB]
- work on collaboration with Florin Rusu [JB]
W14 tasks
=========
continuous integration & unit testing
- now one big task, 2 months
- break unit testing into smaller tasks [JB, done]
Meeting with Florin Rusu
========================
- http://www.ucmerced.edu/faculty/directory/florin-rusu
- innovative approach!
- supports distributed shared scans
- runtime compilation (compiles in gcc) code for in-flight
queries and loads the code dynamically on the server,
can do that in the middle of running queries as new
queries arrive (on chunk boundaries)
- other feature: approximate results with error bounds
- it is academic project, focused on publications/research,
not on path to a production quality product
- debugging looks like a nightmare
- code for known operators templated (M4)
- untrusted user code for aggregations executed on servers
- compilation can take long time (minutes) if many queries,
all inside one large "for" loop, tough for compiler when
optimizations on
- can run up to 64 queries "as one query", beyond that
extra overheads
- the group addresses many of the challenges Qserv
does, so collaborating could be VERY useful
- we can share ideas and/or design
- they can do detailed research
- they can do prototyping
- sharing code is more questionable
- next steps: they will share code with us, we will
determine which parts we want to examine in details
and discuss collaboration [DW, JB]
hackathon planning
==================
-
https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/201311_hackathon
- discuss key components, focus on interfaces
(cap at 1h/component), and
build fake version (1h/component)
- components we identified:
- outside world <--> controller
- query processing module <-- all the rest (big!)
- frontend <--> worker
- query results: worker --> frontend
- interfaces for data loading incl DDL
- accessing metadata
- admin cluster mgmt
- access control system
- work on ~two of the above per day, in smaller groups
- automated integration and unit testing
- 2h
- test by putting tested module into faked environment
- user-friendly communication
- Daniel has some ideas, document on trac
- for the mock-up implementation: a good start
would be to write down a list of the exact
rules that parser should check
- add Serge to that topic [JB, done]
- c++/python
- think more from the perspective: how much of the
existing python an we push to c++?
- primary reason: for debug-ability/coherence,
simpler error reporting
- don't forget to scope how much effort needed
to port from python to c++
- keep in mind that moving to C++ might mean
debugging 100 c++ lines vs 1 line in python
- document what is now done in python [JB]
- plan ~1h for that discussion
- worker complexity
- lay out list of things that can be done
on either side [Daniel]
- building/packaging
- bring Mario/Robert for that discussion [JB]
- data loading
- try implementing mock-up of some parts of
data loader
- add to trac hackathon planning:
- tentative list of outcomes [JB]
Standup mtgs
============
- coordinate input for higher level stand-up mtg
shortly before, possibly afternoon on the
previous day, typically by email
- potentially useful tool that will simplify the
process will be announced/introduced soon
by Mario
Documentation
=============
- nobody seems to have issues with the current structure
Others
======
Send Fabrice info about policies/restrictions
regarding in2p3 contributions *during construction* [JB]
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
