Serge, Daniel, Douglas, Bill, Jacek
phone calls with in2p3
- didn't have one this week
- need a person in charge (setting up audio, announcing if
it is on or off etc)
--> Douglas, will setup readytalk account
- next phone call with in2p3 in 2 weeks (feb 28)
- need to check their availability for monthly
telecons 12pm pacific
- Douglas on vacation next week
qserv milestones in trac
- tweaked as discussed last week, and tickets reassigned
config files
- on worker there is no configuration file
- only xrootd configuration
- no way to parse it, have to parse it manually
- found from Andy that we can pass parameters to
plugin used by xrootd through command line
- this should be sufficient
xrootd client rewriting / APIs
- Daniel sent proposal to Andy, Andy has counter proposal,
discussions will continue later today
branches for qserv
- try to keep ticket branch clean
- do the most dirty work on private branch to not pollute
ticket branch so that commit history looks clean
- master/next branches for qserv?
- Coordination is simple for us, cutting releases is simple
- will stay with just the master branch, no need for "next"
Jacek working on integrating metadata with qserv
loading W13
- Douglas still debugging loading W13_1K data set
- issue: qserv thinks there is no data in partitions
that do have data
- observation: it is still _very_ hard to load data
if we load new data set with new schema etc, even
if it is a tiny amount of data
- having the metadata will help!
logging
- suggestion: setup trac page with wishlist what we want
to have logged in the log file [Douglas]
Apis for duplicator/partitioner that will help install scripts
- file with options is better, command line might run into limit
multi-qserv on the same machine
- Douglas has setup 2 qserv installations on
one machine (W13 and w13_1K)
- issue found: port for xmlrps hardcoded in lua (7080)
- proposed solution: generate correct lua script from
a template on the master during installation
--> create ticket [Douglas]
also...
- qserv master and proxy are started separately,
a thought: qserv master should manage proxy/start it
10k chunks on yilis for concurrency testing
- 1 TB on 120 machines, short on space
- only need object table, that helps
- ok to truncate part of each chunk for this test
in duplicator/partitioner: want option "sample only 10% of input data"
- especially useful in partitioner
increasing density:easily doable thanks to the fact
partitioning now relies on htm, just need to use
one-level-deeper htm ids consistently when populating data
partitioner/duplicator
- few more days to finish partitioner/duplicator
- redoing merge/sort
what if large input data / won't fit on machine?
- fully distribute, run index building in parallel, shuffle
to consolidate data for each triangle on each node
(this is needed because ids get remap and need to be
redistributed to be consistent)
- shuffle: python that fetches data through ssh
- might be slow, but at least will finish
- check with in2p3 what network the 300 nodes have (1Gbit)?
- could rely on one slow nsf drive,
- not worth, still need to move partitions to correct places
- (we should allow generating data on nodes that are
not destination nodes)
data distribution
- need to start thinking about these issues
- eg tracking which node is serving which data, replication,
what if one replica down when data loaded etc, checking
health of all chunks etc
--> start trac page [Jacek]
better error reporting
- Bill has plumbing writen by Daniel
- issues with setting up qserv based on latest commit,
related to swig interface, will debug offline w/Daniel
post processing to load from pipeline to qserv
- match tables are produced pretty late by the pipelines
- if pipelines run in parallel and produce data in many
"blobs", it should help
- still, need to have discussion when KT is at our meeting
logging system (from c++) - which one to use?
- xrootd has its own logging system
- lsst logging package - we'd rather not use it
- log4cxx?
- check with KT
Next meeting in a week
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|