 |
 |
Jacek, (+Daniel) On Oct 25, 2013, at 4:54 PM, Jacek Becla wrote: > In that case, I think we will need to use ra/decl of these sources, > (our scheme has always been that the location of object drives > partitioning for all its sources), refMatch is the only exception. > This will complicate partitioner. I captured that requirement > on trac at: > > https://dev.lsstcorp.org/trac/wiki/db/Qserv/Partitioning > > Serge - please keep that in mind when improving/redesigning > partitioner. Don't worry. This is a minor detail from the POV of the partitioner. Recall that we already ran into this very situation with the association pipeline prototype, in that the source clustering algorithm it employs doesn't necessarily assign every Source to an Object. In that case, I ended up setting the objectId of the Source to NULL, and copying the source ra/dec to the objectRa,objectDec columns, which is equivalent in spirit to what you propose above. One major piece that is missing from the new partitioner is that it currently expects to see a partitioning position in every record (not an ID). In other words, you currently have to feed it denormalized data. That's partly because I figured it should be easy enough for the pipelines to denormalize for me, and partly because I was too lazy to implement an external index mapping from object ID to partitioning position (or chunk/sub-chunk ID). However, the (not too) long term plan is to lift this requirement. As it turns out, Daniel's got a very nice sounding plan for secondary indexes (and I believe he's on the way to a working implementation) that the master will eventually use to lookup the chunk/sub-chunk ID for an object ID. As soon his work on that lands, I'll wire that into the partitioner, or modify it to suit my needs (*), and denormalization prior to partitioning will no longer be necessary. Serge (*) Daniel - in case you are reading along, my thinking here is that there are some not strictly partitioning related computations we'll want to perform prior to being able to serve up a table. Specifically, we'll want to know the maximum angular separation between each coordinate pair in the table and the partitioning coordinates. Given a spatially restricted query against a non- partitioning coordinate pair, we can than constrain which (sub-) chunks must be looked at to satisfy the query by padding the search region with this maximum angular separation, and then looking at only those (sub-) chunks that overlap this padded region. If the object ID index used by the partitioner is from object ID to (RA, Dec) rather than directly to (sub-)chunk ID, then I can easily roll this computation right into partitioning. Another reason the partitioner might prefer a mapping to (RA,Dec) is that it would allow adding overlap to tables other than Object. I do realize this is not currently in the baseline, but maybe it's worth keeping that option open. > On 10/25/2013 04:37 PM, Kian-Tat Lim wrote: >> Jacek, >> >> One thing that came up: Mario and Robert think that we may have >> Sources that cannot be associated with Objects or DiaObjects (that >> will >> become Objects) or SSObjects. These would most likely be noise, >> so they >> don't want to create new Objects or, worse, ForcedSources for >> them. So >> some Sources might have NULL for both objectId and ssObjectId. This >> doesn't change the current schema, but it's something we should >> look out >> for. ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1