[we skipped the mtg last Thursday, this was a replacement]
Present: Fabrice, Douglas, Bill, Serge, Jacek
Bill/logging
- main uses of logging?
- Debugging
- Probably won't be doing heavy analytics as
companies typically do
- access patterns captured through mysql query log
- which should probably be integrated into
the new central logging:
- eg, mysql can write to to file instead of table,
and logging will take it from there
- don't overdesign logging now!
- avoid strong coupling with qserv, will make it easier
to migrate in the future if we need to
Douglas
- started looking at zookeeper, reading docs
and installing
- not clear yet what it could be used for
- document all lessons learned, in 2 trac pages:
dev log and summary
- btw, kafka uses zookeeper
Fabrice
- implemented start/stop service for qserv, (a.la. init.d)
- have script that loads metadata, handy for testing,
now can test only metadata loading. This all used
to be hardcoded
- working on W13 test data set, loaded metadata,
trying which queries not working
- automated testing is working, but many queries broken
- Daniel working on fixing some of them in the parser
- should easily work with buildbot
- need to check with Robyn (but she is busy with FDR now)
Serge
- is taking one more pass on the big parser ticket
- looking into getting rid of config file for
partitioner and duplicator, (custom config file
format, probably better to swig c++/python, and
talk to qms from python, it will be all simpler)
- next: applying fixes for concurrency
- and after than: getting rid of subchunks
- 2 layers of code that execute sql, probably related
to the new shared scan code
Redesign (qserv core)
---------------------
- lot of old, unused code in qserv, need to clean up
- but also need to look at higher level class redesign
- and even higher. Some of the key open questions:
- how smart/dumb a worker should be?
- perhaps a little smarter?
- What are cons of having smarter worker?
- want workers to be easily disposable
- as we increase interface complexity,
debugging gets harder
Things to redesign on the worker:
- class redesign probably needed, eg queries are tracked
by hash, there no hash collision strategy, nothing ever
gets deleted, interractions for 2 queries that start at
~the same time might get mixed up
- perhaps premerge on worker instead of sending individual
chunk results to master
- perhaps instead of sending individual queries to each
subchunk, send simple template to worker, and worker
distributes to subchunks (this increases worker complexity..)
modularity
- master/worker separation is clean, but
beyond that it quickly gets very murky
- modules we thought would be useful to define:
- query rewriting (this is on master only)
- parsing logic (this is on master only)
- thread pooling (this is needed both on M and W,
probably will need one common module, plus two
implementations: one for M and one for W, but
this needs more thinking)
- sql execution (mainly worker, but there are some
things happening on the master too. Potentially
the logic could be structured similarly to thread
pooling: common plus 2 implementations)
- xrootd glue layer
- merging logic
- c++ geometry (on master, could be used on worker)
- logging (small module, eg for things like turning
to json, the actual logging is called from everywhere)
- metadata interface
- chunk mapping
- perhaps put each module in separate name space and
subdirectory?
- each module should have its own unit testing (perhaps
its own directory for tests)
- look at each class in the source code and try to assign
it to module(s)
--> action item: Bill will look into that, will have
something that we can be used to drive discussion
at the next meeting (this Thursday)
- capture in trac page
- definition of module, where it belongs (m/w/both), etc
Redesign (qserv installation/administration)
--------------------------------------------
- two big things that need attention:
a) packaging
b) use of scons
- modularity is not bad
Packaging
- action item: evaluate eups, makes most sense if Fabrice/Emmanuel
do that. Document in trac
Scons
- evaluate use of scons, it is now heavily used in install,
(and for building qserv)
- scons-based build system for qserv needs major rework
- it would be good to get some advice from some experts
outside of qserv team
- action item: write down what scons is doing / what we
want from scons (Fabrice)
- also, need to get rid of perl scripts
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
