 |
 |
On 02/26/2015 03:46 PM, Fabrice Jammes wrote: > Daniel, > > I have a question between the lines below. > > On 02/26/2015 03:32 PM, Daniel L. Wang wrote: >> On 02/26/2015 01:45 PM, Kian-Tat Lim wrote: >>>> The general case is very expensive (lookup position and chunk for each >>>> position!?), and we are only going to get away with it because our >>>> bulk-loads for ForcedSource will be spatially-restricted. >>> I'm pretty sure that ForcedSource and (final) Object tables will >>> be available at the same time, so the partitioner could look up the >>> coordinates based on objectId using a simple merge. I thought you were >>> already going to have a central objectId-to-chunk (or even subchunk?) >>> index anyway; scanning an input ForcedSource table for all its >>> objectIds >>> and then doing a single query (or at least batching objectIds to reduce >>> the number of queries by an order of magnitude) to get the mappings >>> doesn't sound ridiculous. >> I think we're on the same page. The general case is truly expensive: >> loading arbitrary child table rows requires lookups on the director >> table. We can certainly batch this, but again, if child table rows >> come in an arbitrary order, and don't have the one-to-many >> (object<->forcedsource) relationship, it's really expensive. >> >> But yes, batching should be really effective because of the shape of >> our data, and the patterns in which we produce it. I don't think a >> coordinated multi-table partitioning action is scalable: it means >> that the director and all its child table rows need to be available >> *at the same time*. (Oh, want to add another child table? Oh, I guess >> I need to repartition the director and the other 4 child tables and >> reload them. Uh, no.) >> >> My point is not that we don't know how to scale it, but that the >> processing model is different from what we do now. The current >> partitioner and loader can load director and child tables one at a >> time, without checking existing data, sharing only partitioning >> parameters (stripes/substripes). The catch is that it requires the >> child table to be pre-joined (effectively) to the director table in >> order to have partitioning coordinates. We know this is insufficient. > Using the secondary index (built using Object table) to compute the > chunk of a given Source w.r.t its objectId column would avoid this > join, isn't it? > For example, if a source i has objectId field equal to j, then we can > query the secondary index on objectId=j to get the chunk of the > source, this should work. > Of course we have to build the secondary index prior to this operation. This is effectively a join, no? I'm not suggesting sending a SQL join query into the czar's normal pipeline. But looking up chunkId with the secondary index is an index-only join. I think we might still want to create a smaller lookup table for each batch of child table rows, depending on how fast we can make the full index lookups: Is it faster to do 10 million full-index lookups (on disk), or 100k full-index lookups, create a 100k hash table, and 10 million lookups on the in-memory table? I don't know yet. -Daniel ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1