The eventual goal is for the partitioner to be able to partition a source or forcedSource just by looking up the chunk that the associated object was assigned to (rather than requiring the position of that object to be included in the data). The same objectId-to-chunk lookup table (LUT) used by the czar to optimize query dispatch would ideally be used to do this (and might be a by-product of partitioning the Object table).

Since we will be updating Object constantly, we need to be able to update this LUT quickly (i.e. potentially use a fancier data-structure than what Daniel has been thinking about so far), or regenerate a full objectId-to-chunk LUT rapidly. I believe Daniel has a plan + some prototype code to generate a LUT which is basically a list of (objectId:int64_t, chunkId:int32_t) tuples sorted on objectId (there may be a subChunkId or an extra 4 bytes of padding as well). Such a LUT is expected to hit ~640GB by the end of the mission (40e9 obj * 16 bytes/obj). I think Daniel’s structure is supposed to be able to do lookups with one or 2 seeks, but I’m still worried that this is a pretty large amount of data to generate, and may be problematic for the partitioner even though there would likely be a lot of locality in the actual lookups (I’d expect <<1 seek per Source or ForcedSource). Finally, despite being told that we might all be surfing on vast oceans of RAM, worry-free, by the time we actually have to do this… I’m paranoid. Anyway.

I think we can probably reduce the size of the LUT by quite a lot. As far as I know, objects will be created (at a very high level) by measuring sources on patches of an enormous coadd. I'm fairly sure that the object IDs generated for each patch are sequential (of the form patchId*largeConstant + sequenceNumber). So if we instead store tuples that look like (minObjectId:int64_t, delta:int32_t, chunkId:int32_t) that represent:

then we should be able to cut down the size of the LUT by a factor of (very roughly) min(# of objects per coadd patch, # of objects per Qserv chunk). I think patches will be small relative to chunks, but I don’t know how many objects they will contain. Hopefully hundreds to thousands (K-T?). If that's true, the LUT could fit in the vicinity of 1 GB or RAM or so, and we should be able to make index generation operate at the speed of an Object scan: the czar would ask workers to produce chunk-specific index subsets (a small amount of data), then merge them all in-memory.

We might then consider making the LUT completely ephemeral (always in-memory, and generated on Qserv startup). Qserv startup would include launching an Object scan to generate the index. During this scan, objectId lookups would dispatch to all workers, but afterwards dispatch would be optimized as it currently is.