 |
 |
Hello, On 09/04/2013 04:48 PM, Serge Monkewitz wrote: > >> Some other notes: >> - I added LSST__Source (a copy of LSST__Object) to qservMeta. I >> probably could have soft-linked it to LSST__Object.*, but for now I >> just made a copy off the objectId index table. >> - The fact that the qserv master doesn't recognize Source.objectId as >> special and look up a chunk for it in the objectId index table seems >> like a bug. > > So, it turns out that the current qserv master code sub-chunks the > above join (Daniel - is that expected?). The join code thinks that there are always benefits to subchunk a subchunked table when it is being joined. It is stupid, but at least it provides correct results under some circumstances. I added a TODO to think about this, because it doesn't seem like something that we can design and define in our heads as we write code. You might consider the join predicate (hello, USING() and ON() syntax) and what columns are indexed (hello, metadata) or approximate sizes of chunk and subchunk tables. Sometimes it looks like query analyzer and optimizer problem. It looks easy to get wrong. The code is only starting to realize that there is more than one table that might endure objectId lookups. It only recently allowed for the column to be named something other than "objectId". You can call it a bug. It is in the same class as "not implemented". The whole idea of indexing a database by one column needs some design discussion--for me, no details have been hashed out other than the 'we really need this master or boss table thing' hallway comment. > In particular, I see stuff like this in the worker logs on ccqserv27 > for chunk 12833, which contains the objectId in question: > > Runner running Task: msg: session=19 chunk=12833 db=LSST frag: > q=SELECT s.ra,s.decl FROM LSST.Source_12833 AS > s,Subchunks_LSST_12833.Object_12833_0 AS o WHERE > o.objectId=154062577146538502 AND s.objectId=o.objectId,... > > But I also see the same for a bunch of different chunks, and the same > query shows up on other workers. This is what I would call a missed > opportunity by the master-side query generation logic. But the only > ill effect is that execution isn't as fast as it could be. One of many. Should we gather a list? Could be useful for FDR (list of known ways to improve performance beyond our benchmarks). > If I understand things correctly, this means that the Object sub-chunks created on the fly by qserv workers > will always be empty, so that Source/Object join and Object near-neighbor queries will always come back > with nothing on the IN2P3 cluster. Something must have gone seriously wrong with duplication or > partitioning of the Object table. I think it would be useful to come up with a set of sanity-check queries to run on a worker. The sanity check probably involves the qms, and would be run as a post-load check (not required by qserv worker query execution code). e.g. SELECT count(chunkId), count(subChunkId), count(*) from <chunked table> then make sure all are equal. SELECT count(*) from <chunked table> where chunkId <> <chunk#> then make sure count is 0. -Daniel ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1