Attendees:
Serge, Emmanuel, Fabrice, KT, Daniel, Douglas, Jacek,
(and Jeff listening)
Serge
- partitioner: final testing, will push to master
this Friday, ready for integration w/qserv
- duplicator: will be done end of next week
--> make sample refmatch partitioned data available
to Daniel for testing
- integrate with new parser, integrating with old code
would be a waste of ~2 weeks
- side mtg Serge/Daniel re refmatch integration with
new parser APIs
Daniel
- parser: lots of real work done, building in flexibility
for different partitioning schemes
- useful, eg with htm, might want to build overlaps
on the fly
- testable in a few weeks, tbd if will be ready
for large scale tests late June
- shared scans: test it independently from the large
scale test
- shouldn't have any scalability issues, it is per node
Douglas
- setup for testing concurrency on yillis - still debugging
- working on node mgmt tool
- suggestion: look at paramiko, scidb uses it
Fabrice/Emmanuel
- automatic loader for pt 1.1 and unit testing done,
adding more queries and plugging W13
Jacek
- metadata: all prep work done, transient cache etc,
now doing the core work on changing qserv to
integrate it with metadata
- tested pipeQA on W13 data set
- 2 columns with same name issue --> try aliasing
- new parser should fix that
Exposing qserv to few selected users
- perhaps do it now if there is interest,
just explain the limitations
- better version (partitioned RefMatch, new parser and more)
will not be available for at least another month or two
- refmatch integration depends on new parser now
- polygon based searches:
- implementing through qserv non trivial (query composed
on 3 queries), probably a month of work to implement
code in qserv to support it correctly
- can be rewritten as one query with join, but complex
- might introduce special sql syntax and hide complexity
under the hood of qserv
-- maybe look at adql and adopt
- short term solution: get rid of @poly, use hardcoded value,
just to run pipeQA demonstration
--> Serge will document the example
SHOW command
- could use faked schema in short term
- long term: proxy should query metadata
tool for generating empty chunks
- easy to put together based on existing scripts
- would be good to implement solid version, need short
design discussion Douglas/Daniel
ingest
- code written by KT is only for mysql, qserv should
use something different (new)
- better to use fits tables, not csv
- pipelines will be delivering data in chunks as
produced by individual pipeline processes,
- qserv should ingest whole chunk or nothing.
- implement loading plugin that depends on afw
(because of fits tables) and on qserv loader API
- this is long term, short term just
write design doc
Design documentation
- need to write or improve documentation for many things
prior to the July review
- data distribution
- data ingest
- schema evolution
- provenance
- fault tolerance
- and more...
--> work in smaller groups over the next few weeks/months
on all of these
Views
- only power users get to define views
- should be doable if views used only for very well defined
isolated cases, such as
- hiding columns
- joining similarly partitioned tables (eg source x object)
- joining small dimensions tables (star schema)
- join table that has been split into several tables,
eg Object_core, Object_extras, Object_photoZ etc
Next larger qserv meetings 3rd Thursday of next month.
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
