 |
 |
Serge, > I think we can probably reduce the size of the LUT by quite a lot. As > far as I know, objects will be created (at a very high level) by > measuring sources on patches of an enormous coadd. I'm fairly sure > that the object IDs generated for each patch are sequential (of the > form patchId*largeConstant + sequenceNumber). So if we instead store > tuples that look like (minObjectId:int64_t, delta:int32_t, > chunkId:int32_t) that represent: Yes, range-encoding the lookups should help for LSST (and there will be thousands of Objects per patch), but it's not clear that others' object ids will always be so nicely spatially grouped (e.g. Messier numbers or NGC numbers or even a numeric RA+Dec encoding). > We might then consider making the LUT completely ephemeral (always > in-memory, and generated on Qserv startup). Qserv startup would > include launching an Object scan to generate the index. During this > scan, objectId lookups would dispatch to all workers, but afterwards > dispatch would be optimized as it currently is. So initial point queries would still be fast but overall capacity would be limited until the scan is complete? I'm a little worried about what it would take to bring up a new czar to replace a failed one: copy the LUT from an existing czar, regenerate from the workers, or rescan Object? -- Kian-Tat Lim, LSST Data Management, [log in to unmask] ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1