 |
 |
Hello Jacek and Douglas, About continuous integration, i need to add a small information. First, thanks for reviewing #3004 but #3013 (init.d scripts), which relies on #3004, is still waiting for a review. Could your team review it quickly please ? When this two tickets will be closed, then i steel need to open a last ticket (related to test dataset) in order to have qserv install which can build on continuous integration (the code related to this one is nearly ready but it strongly relies on #3004 and #3013). This can be done at the beginning of this week assuming previous tickets are reviewed. Fabrice On 11/01/2013 02:36 AM, Jacek Becla wrote: > Attendees: > Daniel [DW] > Serge [SMM] > Douglas [DS] > Fabrice [FJ] > Jacek [JB] > > > Agenda: > - quick round-table status report & next steps > - W14 db tasks > - EXTASCID - debrief from mtg, next steps > - hackathon planning > - stand-up meetings > - documentation > > > > Status report > ============= > - data ingest and loading design discussions > [JB, SMM, KT, DW] and documentation [JB] > - W14 planning [JB] > - meeting with Florin Rusu (more below) [JB, DW, KT] > - code cleanup [DS] > - db trac page cleanup [JB] > - playing with Zookeeper [DS] > - work on Qserv modularization [DW, BC] > > > > Next steps (this covers all action items identified at > the meeting, some sprinkled below throughout the notes) > ======================================================= > code cleanup (want it done by next Thu) > - finish #3007 [DS] > - finish #3004, #3013 [FJ + ticket reviewers] > - when the 3 above done, will be ready for integration > with buildbot > - want all changes pushed to master by next Thu > - review Douglas' ticket [DW] > - merge 3 tickets [BC] > - concurrency fixes - put in a ticket, by early > next week [SMM] > - create new tag for Qserv that captures all > useful code lingering around, by next Thu [DS] > > prep for hackathon (all items in this section are > high priority) > - document what is now done in python [JB] > - hackathon planning [JB + input from all] > - user friendly communication and syntax parsing: > Daniel has some ideas, document on trac [DW] > - advance work on modularization and present next > Thu at qserv meeting [DW, BC] > - lay out list of things that can be done > on either worker or master [DW] > - bring Mario/Robert if possible (remotely) for > discussion about building/packaging, during > hackathon week [JB] > - add to trac hackathon planning: "tentative list > of outcomes" [JB] > - take apart wishlists and document each point > raised in appropriate place [JB] > > coding (low priority) > - work on objectId index [DW] > - join syntax improvements [DW] > - small tweaks to shared scan ticket [DW] > - later, after next week, when we have the new > Qserv tag: cleanup all Qserv tickets [JB?] > - follow up with Florin Rusu, check his code [DW] > > unclassified > - talk to Robyn about integrating with buildbot > sometime later next week [JB] > - send Fabrice info about policies/restrictions > regarding in2p3 contributions *during > construction* [JB] > - work on collaboration with Florin Rusu [JB] > > > W14 tasks > ========= > continuous integration & unit testing > - now one big task, 2 months > - break unit testing into smaller tasks [JB, done] > > > > Meeting with Florin Rusu > ======================== > - http://www.ucmerced.edu/faculty/directory/florin-rusu > - innovative approach! > - supports distributed shared scans > - runtime compilation (compiles in gcc) code for in-flight > queries and loads the code dynamically on the server, > can do that in the middle of running queries as new > queries arrive (on chunk boundaries) > - other feature: approximate results with error bounds > > - it is academic project, focused on publications/research, > not on path to a production quality product > - debugging looks like a nightmare > - code for known operators templated (M4) > - untrusted user code for aggregations executed on servers > - compilation can take long time (minutes) if many queries, > all inside one large "for" loop, tough for compiler when > optimizations on > - can run up to 64 queries "as one query", beyond that > extra overheads > > - the group addresses many of the challenges Qserv > does, so collaborating could be VERY useful > - we can share ideas and/or design > - they can do detailed research > - they can do prototyping > - sharing code is more questionable > - next steps: they will share code with us, we will > determine which parts we want to examine in details > and discuss collaboration [DW, JB] > > > > hackathon planning > ================== > - > https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/201311_hackathon > - discuss key components, focus on interfaces > (cap at 1h/component), and > build fake version (1h/component) > - components we identified: > - outside world <--> controller > - query processing module <-- all the rest (big!) > - frontend <--> worker > - query results: worker --> frontend > - interfaces for data loading incl DDL > - accessing metadata > - admin cluster mgmt > - access control system > - work on ~two of the above per day, in smaller groups > > - automated integration and unit testing > - 2h > - test by putting tested module into faked environment > > - user-friendly communication > - Daniel has some ideas, document on trac > - for the mock-up implementation: a good start > would be to write down a list of the exact > rules that parser should check > - add Serge to that topic [JB, done] > > - c++/python > - think more from the perspective: how much of the > existing python an we push to c++? > - primary reason: for debug-ability/coherence, > simpler error reporting > - don't forget to scope how much effort needed > to port from python to c++ > - keep in mind that moving to C++ might mean > debugging 100 c++ lines vs 1 line in python > - document what is now done in python [JB] > - plan ~1h for that discussion > > - worker complexity > - lay out list of things that can be done > on either side [Daniel] > > - building/packaging > - bring Mario/Robert for that discussion [JB] > > - data loading > - try implementing mock-up of some parts of > data loader > > - add to trac hackathon planning: > - tentative list of outcomes [JB] > > > Standup mtgs > ============ > - coordinate input for higher level stand-up mtg > shortly before, possibly afternoon on the > previous day, typically by email > - potentially useful tool that will simplify the > process will be announced/introduced soon > by Mario > > > > Documentation > ============= > - nobody seems to have issues with the current structure > > > Others > ====== > Send Fabrice info about policies/restrictions > regarding in2p3 contributions *during construction* [JB] > > > Jacek > > _______________________________________________ > LSST-data mailing list > [log in to unmask] > http://listserv.lsstcorp.org/mailman/listinfo/lsst-data ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1