Hello Qserv clan,

On my run, I was thinking about the subchunking and overlap problem, and 
the dec-stripe solution, and I think I've come up with an in-between 
solution.

One of the reasons that we just stored overlap and then queried by using 
joining against subchunk+overlap, was that we didn't want to compute the 
exact overlap region for each query, for each subchunk. How about this:

* store subchunks, but not overlap.
* query against subchunk + adjacent subchunks. Effectively, this gives 
us overlap ~= subchunk width.
* computing this may have some noticeable cost if the czar has to do it 
(full scan of 10k chunks each with 200 subchunks = computation of 2 
million adjacency subchunks), so maybe we can push it to the workers, 
where it can be computed almost for free (actually, we can cache it on 
each worker)..
* subchunks are on-the-fly, so we save computation of overlap subchunks 
completely, and build half the temp tables as before
* Workaround subchunks from adjacent chunks by storing them using 
virtual subchunk numbers.
* This requires somewhat more code and complexity, but also eliminates 
the previous overlap management, so the net cost/complexity increase is 
low(?).


Another different idea: We build subchunk tables on the fly because the 
mysql optimizer is too stupid to use a "subchunkid=X" condition in the 
WHERE clause to its fullest effect. Did we try coaxing it by using a 
subquery?

i.e., SELECT o1.blah, o2.blah FROM (SELECT ... FROM Object_N WHERE 
subchunkid=X) as o1, (SELECT ... FROM Object_N WHERE subchunkid = X) AS 
o2 WHERE...
instead of
SELECT o1.blah, o2.blah FROM Object_N AS o1, Object_N AS o2 WHERE 
subchunkid = X AND ...;

Have a great weekend, and thanks for the great week!
-Daniel

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1