Print

---

Attendees: Daniel, AndyS, Fabrice, Bill, Serge, Jacek

Agenda:
  - round table status report
  - S14 planning
  - random topics


Round table status report
=========================

Fabrice:
  - packaged qserv through eupsPkg, works
  - relies on distribution server at in2p3,
    public access ok, so new code can be
    tested on non-in2p3 servers
  - making code more robust now
  - working on configuration, will be in qserv/admin,
    will generate template config file, (which will
    need to be tweaked by an administrator)
    - this will be done through scons, as before
    - replaces old scripts written by Douglas, Emmanuel
      and Fabrice
  - the code in ticket #3100
  - non-trivial scons work
  - hopefully all finished in 1-2 weeks
  - it is a useful iteration to do
  - will revisit the whole configuration design after
    Fabrice done with this iteration
    (e.g., integrating with Zookeeper, bootstraping nodes etc)


Bill:
  - created a new module (called protolog for now) that
    uses log4cxx
  - have swig interface
  - all critical pieces in place
  - logging works in both c++ and python layer
  - still needs cleanup
  - most logging code not migrated
    - convert selected pieces to stress the new logging code
  - added option: file name/source code line number on/off
     - would be useful to have switch at different level:
       e.g. globally, per logger
  - compatible with what was discussed broadly few months
    ago, documented here:
     https://dev.lsstcorp.org/trac/wiki/db/Qserv/Logging
  - will discuss with broader DM in about 2 weeks
  - AndyS has useful suggestions, will post comments


Daniel:
  - worked on join syntax support
  - 3 way join works, doing code cleanup now
    - next: testing if subchunking code works
  - Serge should review, early next week


Serge:
  - need few more days to finish WISE work


Jacek:
  - have all dev env setup with the latest code,
    based on Fabrice eupsPkg + new qserv build
    system
    - on ubuntu 12.04
  - integrating css with Qserv, including dummy
    (zookeeper-free) implementation for testing
  - XLDB: workshops prep, long-term strategy
  - deciding on pre-commissioning milestones
     - architecture at 24 months
     - key dependencies at 18 months
     - interfaces at 12 months
     - versions at 6 months


S14 planning
============
  [we have a draft spreadsheet capturing Qserv/DB tasks
  for S14, it will be circulated in a week or two.
  Random points discussed are covered below]

  - configuration vs zookeeper:
    - configuration often more stable, committed in git,
      info in zookeeper-type metadata more dynamic
  - chef, puppet, few others to consider
    --> create trac page

data partitioning, including research to avoid subchunking
  - yes, keep in S14

worker scheduler design
  - some of that cleanup will be done during query result cleanup
  - real redesign - not in S14 (well, maybe, it is a stretch goal)

user query mgmt iface design
  - not S14, but revisit later, after CSS stable

data distribution design
  - keep, need to define scope of what we want to achieve in S14

L3
  - not S14

reference test server using new xrd
  - keep in S14, mostly for AndyH

parser - v2
  - that includes parser and framework for query representation
  - parser in reasonable state as is,
    - ok to go into construction with what we have

fault tolerance
  - nothing for S14

front end rearchitect
  - that includes proxy, down to xrootd client
    (mysqld, python, zookeeper, ...)
  - yes, spend some time rethinking current architecture,
    what are the problems, how new system would look like
   --> need to document

secondary indexes
  - can go into constr with what we have
  - dynamic update complicates the design

tools for cluster coordination (implementation)
  - nothing serious in S14, but good to play with it soon

geometry
  - it is a straight port from python to c++
  - need to think about depending on eigen vs using
    homegrown code (which Serge already has)
  - can go into constr with what we have
  - but super ugly what we have now, so keep in S14


Random topics
=============

Multi-platform support:
  - would rather not clatter the build scripts (at least not yet)
  - don't want to officially support multiple platforms
  - if easy to get it running on Ubuntu, worth spending some time
     - not sure if lsst have standard way to check which platform
       we are on (from within scons)

mysql user conf
  - a chance to talk to mysql gurus
  - Daniel going

SciSQL for MariaDB
   - Serge will look into it

SciSQL
- move scisql to github, don't want to deal with bazaar

design/hackathon week
  - 4/28-5/3 works for AndyS, Daniel, Fabrice, Jacek, Serge, KT
  - need to check with Bill and AndyH

embedding mysql into qserv
  - instead of running as separate daemon
    - for worker and/or master
  --> open trac page
  - think about, maybe discuss at the design/hackathon week

Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1