Quick summary of what we (AndyS, Nate and me) discussed.
We will discuss it more at the Qserv meeting…
* Zookeeper is not great for quick and frequent updates, so implementing
things like synchronizing access to resources such as table state/metadata
in zookeeper is questionable
* if we combine it with fact that watchers are not reliable and with the pains
with c++ bindings, we’d basically want to …. get rid of zookeeper
Proposal
* push CSS metadata into transactional mysql
* register every query in query metadata, including interactive
* change Facade API: new API would only have “fetch all metadata for a given
table”
* KVInterface - stays as is (granularity: key/value). Add support for updates
(Eg we need to set flags used to sync access to tables and that’d be done
through KVInterface)
* implement new MySQLKvInterface
* need to think how to store CSS info internally in mysql. Pack as JSON?
* high availability: standby replica(s) + auto-fail over. Two modes
1) Don’t handle new queries while fail over is happening, or
2) we could maintain a stand-by snapshot in czar, and work off the snapshot
while fail over is happening
* need a flag in mysql per table to synchronize access
- process that wants to delete a table would lock it (exclusive lock)
- czar would lock it (shared read lock) when it starts processing new query
* Need python interface for qservAdmin type things, Could wrap the C++ one
I hope that is enough to plan it in your heads. Think about it over the next
day or so, and we will discuss more on Wed.
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
