Attendees:
Serge, Emmanuel, Fabrice, KT, Daniel, Douglas, Jacek,
(and Jeff listening)


Serge
   - partitioner: final testing, will push to master
     this Friday, ready for integration w/qserv
   - duplicator: will be done end of next week
   --> make sample refmatch partitioned data available
     to Daniel for testing
   - integrate with new parser, integrating with old code
     would be a waste of ~2 weeks
   - side mtg Serge/Daniel re refmatch integration with
     new parser APIs

Daniel
   - parser: lots of real work done, building in flexibility
     for different partitioning schemes
     - useful, eg with htm, might want to build overlaps
       on the fly
     - testable in a few weeks, tbd if will be ready
       for large scale tests late June

   - shared scans: test it independently from the large
     scale test
      - shouldn't have any scalability issues, it is per node

Douglas
   - setup for testing concurrency on yillis - still debugging
   - working on node mgmt tool
     - suggestion: look at paramiko, scidb uses it

Fabrice/Emmanuel
   - automatic loader for pt 1.1 and unit testing done,
     adding more queries and plugging W13

Jacek
   - metadata: all prep work done, transient cache etc,
     now doing the core work on changing qserv to
     integrate it with metadata
   - tested pipeQA on W13 data set
   - 2 columns with same name issue --> try aliasing
     - new parser should fix that

Exposing qserv to few selected users
   - perhaps do it now if there is interest,
     just explain the limitations
   - better version (partitioned RefMatch, new parser and more)
     will not be available for at least another month or two
      - refmatch integration depends on new parser now
   - polygon based searches:
     - implementing through qserv non trivial (query composed
       on 3 queries), probably a month of work to implement
       code in qserv to support it correctly
     - can be rewritten as one query with join, but complex
     - might introduce special sql syntax and hide complexity
       under the hood of qserv
       -- maybe look at adql and adopt
     - short term solution: get rid of @poly, use hardcoded value,
       just to run pipeQA demonstration
       --> Serge will document the example

SHOW command
   - could use faked schema in short term
   - long term: proxy should query metadata

tool for generating empty chunks
   - easy to put together based on existing scripts
   - would be good to implement solid version, need short
     design discussion Douglas/Daniel

ingest
   - code written by KT is only for mysql, qserv should
     use something different (new)
   - better to use fits tables, not csv
   - pipelines will be delivering data in chunks as
     produced by individual pipeline processes,
      - qserv should ingest whole chunk or nothing.
   - implement loading plugin that depends on afw
     (because of fits tables) and on qserv loader API
     - this is long term, short term just
       write design doc

Design documentation
   - need to write or improve documentation for many things
     prior to the July review
     - data distribution
     - data ingest
     - schema evolution
     - provenance
     - fault tolerance
     - and more...
   --> work in smaller groups over the next few weeks/months
       on all of these

Views
   - only power users get to define views
   - should be doable if views used only for very well defined
     isolated cases, such as
      - hiding columns
      - joining similarly partitioned tables (eg source x object)
      - joining small dimensions tables (star schema)
      - join table that has been split into several tables,
        eg Object_core, Object_extras, Object_photoZ etc

Next larger qserv meetings 3rd Thursday of next month.

Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1