Attendees: Daniel, AndyS, Fabrice, Bill, Serge, Jacek
Agenda:
- round table status report
- S14 planning
- random topics
Round table status report
=========================
Fabrice:
- packaged qserv through eupsPkg, works
- relies on distribution server at in2p3,
public access ok, so new code can be
tested on non-in2p3 servers
- making code more robust now
- working on configuration, will be in qserv/admin,
will generate template config file, (which will
need to be tweaked by an administrator)
- this will be done through scons, as before
- replaces old scripts written by Douglas, Emmanuel
and Fabrice
- the code in ticket #3100
- non-trivial scons work
- hopefully all finished in 1-2 weeks
- it is a useful iteration to do
- will revisit the whole configuration design after
Fabrice done with this iteration
(e.g., integrating with Zookeeper, bootstraping nodes etc)
Bill:
- created a new module (called protolog for now) that
uses log4cxx
- have swig interface
- all critical pieces in place
- logging works in both c++ and python layer
- still needs cleanup
- most logging code not migrated
- convert selected pieces to stress the new logging code
- added option: file name/source code line number on/off
- would be useful to have switch at different level:
e.g. globally, per logger
- compatible with what was discussed broadly few months
ago, documented here:
https://dev.lsstcorp.org/trac/wiki/db/Qserv/Logging
- will discuss with broader DM in about 2 weeks
- AndyS has useful suggestions, will post comments
Daniel:
- worked on join syntax support
- 3 way join works, doing code cleanup now
- next: testing if subchunking code works
- Serge should review, early next week
Serge:
- need few more days to finish WISE work
Jacek:
- have all dev env setup with the latest code,
based on Fabrice eupsPkg + new qserv build
system
- on ubuntu 12.04
- integrating css with Qserv, including dummy
(zookeeper-free) implementation for testing
- XLDB: workshops prep, long-term strategy
- deciding on pre-commissioning milestones
- architecture at 24 months
- key dependencies at 18 months
- interfaces at 12 months
- versions at 6 months
S14 planning
============
[we have a draft spreadsheet capturing Qserv/DB tasks
for S14, it will be circulated in a week or two.
Random points discussed are covered below]
- configuration vs zookeeper:
- configuration often more stable, committed in git,
info in zookeeper-type metadata more dynamic
- chef, puppet, few others to consider
--> create trac page
data partitioning, including research to avoid subchunking
- yes, keep in S14
worker scheduler design
- some of that cleanup will be done during query result cleanup
- real redesign - not in S14 (well, maybe, it is a stretch goal)
user query mgmt iface design
- not S14, but revisit later, after CSS stable
data distribution design
- keep, need to define scope of what we want to achieve in S14
L3
- not S14
reference test server using new xrd
- keep in S14, mostly for AndyH
parser - v2
- that includes parser and framework for query representation
- parser in reasonable state as is,
- ok to go into construction with what we have
fault tolerance
- nothing for S14
front end rearchitect
- that includes proxy, down to xrootd client
(mysqld, python, zookeeper, ...)
- yes, spend some time rethinking current architecture,
what are the problems, how new system would look like
--> need to document
secondary indexes
- can go into constr with what we have
- dynamic update complicates the design
tools for cluster coordination (implementation)
- nothing serious in S14, but good to play with it soon
geometry
- it is a straight port from python to c++
- need to think about depending on eigen vs using
homegrown code (which Serge already has)
- can go into constr with what we have
- but super ugly what we have now, so keep in S14
Random topics
=============
Multi-platform support:
- would rather not clatter the build scripts (at least not yet)
- don't want to officially support multiple platforms
- if easy to get it running on Ubuntu, worth spending some time
- not sure if lsst have standard way to check which platform
we are on (from within scons)
mysql user conf
- a chance to talk to mysql gurus
- Daniel going
SciSQL for MariaDB
- Serge will look into it
SciSQL
- move scisql to github, don't want to deal with bazaar
design/hackathon week
- 4/28-5/3 works for AndyS, Daniel, Fabrice, Jacek, Serge, KT
- need to check with Bill and AndyH
embedding mysql into qserv
- instead of running as separate daemon
- for worker and/or master
--> open trac page
- think about, maybe discuss at the design/hackathon week
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|