Hey, sorry I missed today's meeting. I was utterly mistaken in believing
that today's meeting would be at 1:15pm (I don't know why), and thus
didn't read the reminder email for the 9:15pm meeting (which would have
clued me in).
Comments inline:
On 10/15/2013 01:14 PM, Jacek Becla wrote:
> Bill/logging
> - main uses of logging?
> - Debugging
> - Probably won't be doing heavy analytics as
> companies typically do
> - access patterns captured through mysql query log
> - which should probably be integrated into
> the new central logging:
> - eg, mysql can write to to file instead of table,
> and logging will take it from there
> - don't overdesign logging now!
> - avoid strong coupling with qserv, will make it easier
> to migrate in the future if we need to
Please have a look at pex_logging. I will, as well. While its backend
may be ugly, the user interface could be close to what we really want.
> Douglas
> - started looking at zookeeper, reading docs
> and installing
> - not clear yet what it could be used for
> - document all lessons learned, in 2 trac pages:
> dev log and summary
> - btw, kafka uses zookeeper
I highly recommend reading the USENIX paper on zookeeper
https://www.usenix.org/event/usenix10/tech/full_*papers*/Hunt.pdf
A write-mostly-read, distributed, persistent, highly-available data
store seems eminently useful for making sure that qserv nodes agree on
things like configuration and everything in the qms. It eliminates the
single-point-of-failure that the qms represents. All nodes can watch for
changes, so that everyone agrees on a single consistent, coherent state.
I'd say that's useful. It's not for query flow, because state changes
are kind of slow (10k writes per second) and the entire state is
replicated. Zookeeper survives as long as >50% of its nodes are up.
> Fabrice
>
> - automated testing is working, but many queries broken
Some of those tests will be broken for quite some time. Hence, let's
make sure the test report shows which tests passed and failed, because
collapsing lots of passed test plus one failure into a single red
"FAILED" report is not useful. Maybe this is already fixed?
> Serge
> - is taking one more pass on the big parser ticket
What? It's merged.
But, opening a ticket to implement some of the reworking that Serge
suggested for the parser... that I would really support.
> - 2 layers of code that execute sql, probably related
> to the new shared scan code
This 2-headed sql access bothers me. But it's basically Python-layer vs
C++ layer, and our mythical ideal system will only access via C++
(technically, C). There is a bit of sql access done for pushing back
messages for the user that can be done in C++ and SWIGed.
> Redesign (qserv core)
> ---------------------
> - lot of old, unused code in qserv, need to clean up
Completely agree.
> - but also need to look at higher level class redesign
Yes!
> - and even higher. Some of the key open questions:
> - how smart/dumb a worker should be?
> - perhaps a little smarter?
The worker should know as little as possible, but it should know a lot
about what it owns. This is more than it knows now.
It should not know about the state of other workers. I would like the
worker to do zero parsing. If it needs to derive characteristics/stats
from the query, the master should pass those annotations down.
> - What are cons of having smarter worker?
Coordination in maintaining a consistent state.
More decisions made by the worker means a more complex system, but
possibly one that scales better. More decisions by the worker mean that
worker behavior is more complex to understand. Dumb workers are easy to
understand and lead to a simpler system.
> - want workers to be easily disposable
> - as we increase interface complexity,
> debugging gets harder
The more responsibility a worker has, it is potentially more of a hassle
to failover. Read-only-and-stateless has a certain charm.
> Things to redesign on the worker:
> - class redesign probably needed, eg queries are tracked
> by hash, there no hash collision strategy, nothing ever
> gets deleted, interractions for 2 queries that start at
> ~the same time might get mixed up
We just need to implement the results management that we (or I) have put
off implementing. Eventually, we need to cache results, and even if we
don't need it now, a basic implementation would handle the double query
case and just reuse results.
> - perhaps premerge on worker instead of sending individual
> chunk results to master
I started on this, but it really should wait until we do the result
protocol implementation that eliminates mysqldump.
> - perhaps instead of sending individual queries to each
> subchunk, send simple template to worker, and worker
> distributes to subchunks (this increases worker complexity..)
This complexity is fine. I think I discussed this with Serge a while
back, and we came up with a toy templating syntax whose details I
promptly forgot.
> modularity
> - master/worker separation is clean, but
> beyond that it quickly gets very murky
> - modules we thought would be useful to define:
> - query rewriting (this is on master only)
Good.
> - parsing logic (this is on master only)
This is small.
- query intermediate format (both parser and rewriter depend on it, but
do not depend on each other)
> - thread pooling (this is needed both on M and W,
> probably will need one common module, plus two
> implementations: one for M and one for W, but
> this needs more thinking)
Sure, yes.
> - sql execution (mainly worker, but there are some
> things happening on the master too. Potentially
> the logic could be structured similarly to thread
> pooling: common plus 2 implementations)
Right, I think it's in common/ right now, but the python parts ignore it.
> - xrootd glue layer
And...
query execution, scheduler, admission control on master and worker (6
modules)
> - merging logic
> - c++ geometry (on master, could be used on worker)
hooray!
> - logging (small module, eg for things like turning
> to json, the actual logging is called from everywhere)
I would really like to have logging done in a way that depends on
nothing, like scisql, and have compile-time bindings for qserv+lsst
specifics. This would make it easy to share with LSST and the interwebs.
> - metadata interface
Maybe some sort of abstraction for system state and metadata that
shields code from a zookeeper-like system? Kind of a hassle, because the
zookeeper (and similar) have very simple interfaces, but there are
arguments.
> - chunk mapping
I sort of tried to do this, but the chunking concept needs some design.
I think K-T finds that all possible qserv partitioning schemes will have
chunk (and subChunk), but the allocation of chunkId would differ.
> - perhaps put each module in separate name space and
> subdirectory?
Probably. And make .a files that get linked into the proper combination
for a .so on master and worker.
> - each module should have its own unit testing (perhaps
> its own directory for tests)
arguable in general, but a good idea for a significant fraction. I think
the organization is dependent on the balance of complexity and
convenience in our mythical scons layout.
> Scons
>
> - scons-based build system for qserv needs major rework
Clean-sheet implementation is desired. Whoever is involved should take a
look at examples of software using scons and cmake (for perspective, at
least). The build system should be separated from the installer, and
should know how to build the software and place it into an installed
location, knowing only the locations of the dependencies and the desired
installed location. It shouldn't setup the software, but may place
sample configuration in a reasonable place.
> - it would be good to get some advice from some experts
> outside of qserv team
In the event that they don't have time, I think the organization of some
of the afw classes might be good data points--there are single packages
with different modules that have their own unit tests.
Hope this helps! Again, sorry I missed this morning.
-Daniel
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
