Attendees: Daniel, Serge, Jacek

* recent update to redhat 6.3 broke boost (1.41) on slac machines.
   - related Red Hat bugzilla case: #908774
   - ended up downgrading offending packages to get
     back to working version
   - redhat is downplaying the importance of this fix, but
     other labs are starting to join us and complain too

* weekly status report
  --> need to update more regularly, many qserv tasks
    not updated

* qserv review
  - tentatively planning for late July

* need to plan & document our plans re
   schema evolution and provenance
  - started trac page about schema evolution
    https://dev.lsstcorp.org/trac/wiki/db/SchemaEvolution
  - provenance harder!
  - dedicated phone call to discuss these issues
    Tue Mar 5 at 10:00am pacific

* upgrading boost to boot 1.53?
  - Serge would find some new features handy
  - no, too aggressive, better to rely on stable,
    mainstream packages that come with OS

* extra feature for partitioner: auto-detecting
   csv separator type
   - would be nice to auto-detect in loader if
     input rows are tab or comma separated,
   - requested/suggested by in2p3
   - ok to assume all files from given group have
     same separator? Yes
   --> defer adding new features like this until ongoing
       work on partitioner finished

* another extra feature for partitioner:
   read field names from the first line of file
  - would be nice to have
  - implement later, low priority

* to keep things simpler, we should require input data
   to perfectly match schema of table we load into
    - if realignment needed, pre-process input data

* should partitioner be able to reorder columns?
  - don't over-complicate for now

* should partitioner be able to drop columns from
   input data
  - needed to drop _chunkId column from our input
    data from pt1.2
  - don't over-complicate, clean up in separate
    clean up step, not in partitioner

* we want partitioner/duplicator to support
   sampling (eg produce 10% of what could be produced)
  - synchronizing ids between tables would be useful
  - implement in duplicator, not partitioner
    - already have ids in memory in duplicator,
      so it is easy

refmatch
  - patching qserv to deal with refmatch will
    take ~1 week
  - do shortly after partitioner ready


how should refmatch be treated in metadata
  - just add flag "isMatchTable"
  - should we also store info which tables are
    related to a given match table?
    - particularly useful if we want to manage
      multiple clusters of match tables, eg
     2mass2object, sdss2object, etc...
     - would be nice, for later


DC_W13_Stripe82_40deg does not have index
on RunDeepForcedSource.objectId
  - DC_W13_Stripe82 does
  - need to add index building to scripts used
    for production so that things are consistent
    and not forgotten
    - a mess... ingest for either Source or
      ForcedSource was rewritten, not sure which
      code is exercised, need to talk to KT


myisam
- can't sort by random column, only by index


Daniel redoing logging system for worker as part
of export path work

Jacek working on integrating metadata with qserv.

Helped Dave Monet, no need to use qserv for that.

Still want to try performance for differently
sorted data
  - managed to kill mysqld on lsst10 while sorting
    a copy of stripe82_40 deg last night...


Thanks,
Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1