K-T,

On Feb 7, 2013, at 5:35 PM, Kian-Tat Lim <[log in to unmask]> wrote:

>> - bigger issue: apps code does not produce all columns
>>   we need (eg ra/decl for ForcedSources not there), so
>>   we need to post-process data they produce
>> - would be nice if pipelines code would do that
>>    --> talk to KT about it
> 
> 	I don't think this is going to happen easily.  Robert would like
> to work entirely in x/y space; RA/dec is an add-on convenience.  While
> it's not totally unreasonable to expect that the Object x/y and Exposure
> WCS (which then allows the computation of the Object RA/dec) will be
> available when ForcedSources are computed, I think there are going to be
> other tables that are more difficult to do this for (e.g. DiaSource,
> where the mapping to Object will not be known until after the DiaSources
> are generated).
> 
> 	I think it would be better for you to plan to load per-Object
> RA/dec (and objectId, of course) into the partitioner and also be
> prepared to load Object-to-Foo match tables while loading the Foo table.

> 	We can do all this computation/pre-join as a separate pass, but
> someone is going to have to do it, and it might as well be the qserv
> partitioner/loader.

As long as this information gets produced in fairly coherent fashion (i.e. I can do the join on small sets of related files in memory) that's OK, but it will not be in this first cut of the partitioner. For now we can just include the columns we need by dumping from the W13 production database.

Is that a safe assumption though? If I have to deal with joins that involve 10s of terabytes of ASCII files though, this is going to be non-trivial to do well.

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1