Serge, Daniel, Douglas, Bill, Jacek


phone calls with in2p3
  - didn't have one this week
  - need a person in charge (setting up audio, announcing if
    it is on or off etc)
  --> Douglas, will setup readytalk account
  - next phone call with in2p3 in 2 weeks (feb 28)
     - need to check their availability for monthly
       telecons 12pm pacific
  - Douglas on vacation next week


qserv milestones in trac
  - tweaked as discussed last week, and tickets reassigned


config files
  - on worker there is no configuration file
  - only xrootd configuration
     - no way to parse it, have to parse it manually
  - found from Andy that we can pass parameters to
    plugin used by xrootd through command line
     - this should be sufficient

xrootd client rewriting / APIs
  - Daniel sent proposal to Andy, Andy has counter proposal,
    discussions will continue later today

branches for qserv
  - try to keep ticket branch clean
  - do the most dirty work on private branch to not pollute
    ticket branch so that commit history looks clean
  - master/next branches for qserv?
    - Coordination is simple for us, cutting releases is simple
    -  will stay with just the master branch, no need for "next"

Jacek working on integrating metadata with qserv

loading W13
  - Douglas still debugging loading W13_1K data set
  - issue: qserv thinks there is no data in partitions
    that do have data
  - observation: it is still _very_ hard to load data
    if we load new data set with new schema etc, even
    if it is a tiny amount of data
  - having the metadata will help!


logging
  - suggestion: setup trac page with wishlist what we want
    to have logged in the log file [Douglas]


Apis for duplicator/partitioner that will help install scripts
    - file with options is better, command line might run into limit


multi-qserv on the same machine
  - Douglas has setup 2 qserv installations on
    one machine (W13 and w13_1K)
  - issue found: port for xmlrps hardcoded in lua (7080)
  - proposed solution: generate correct lua script from
    a template on the master during installation
    --> create ticket [Douglas]
also...
  - qserv master and proxy are started separately,
    a thought: qserv master should manage proxy/start it


10k chunks on yilis for concurrency testing
  - 1 TB on 120 machines, short on space
  - only need object table, that helps
  - ok to truncate part of each chunk for this test


in duplicator/partitioner: want option "sample only 10% of input data"
  - especially useful in partitioner


increasing density:easily doable thanks to the fact
partitioning now relies on htm, just need to use
one-level-deeper htm ids consistently when populating data


partitioner/duplicator
  - few more days to finish partitioner/duplicator
  - redoing merge/sort


what if large input data / won't fit on machine?
  - fully distribute, run index building in parallel, shuffle
    to consolidate data for each triangle on each node
    (this is needed because ids get remap and need to be
    redistributed to be consistent)
   - shuffle: python that fetches data through ssh
      - might be slow, but at least will finish
          - check with in2p3 what network the 300 nodes have (1Gbit)?
   - could rely on one slow nsf drive,
     - not worth, still need to move partitions to correct places
     - (we should allow generating data on nodes that are
       not destination nodes)


data distribution
  - need to start thinking about these issues
    - eg tracking which node is serving which data, replication,
      what if one replica down when data loaded etc, checking
      health of all chunks etc
    --> start trac page [Jacek]


better error reporting
  - Bill has plumbing writen by Daniel
  - issues with setting up qserv based on latest commit,
    related to swig interface, will debug offline w/Daniel


post processing to load from pipeline to qserv
  - match tables are produced pretty late by the pipelines
  - if pipelines run in parallel and produce data in many
    "blobs", it should help
  - still, need to have discussion when KT is at our meeting


logging system (from c++) - which one to use?
  - xrootd has its own logging system
  - lsst logging package - we'd rather not use it
  - log4cxx?
  - check with KT

Next meeting in a week

Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1