I added an epic

https://jira.lsstcorp.org/browse/DM-2187

to capture loading tables with no ra/decl

Jacek



On 02/26/2015 12:39 PM, Daniel L. Wang wrote:
> I would like to note that the current system requires the extra raObject
> and declObject columns in ForcedSource, so that table's size will be
> proportionally larger than it would be in production. Do we actually
> have ForcedSource data? I think it would be useful to add some
> artificial data to one of our test cases (generate 10 randomly generated
> ForcedSource rows per Object row in case01), so we can actually run that
> test.
>
> The code to load child table rows that lack director positioning (other
> than the director's primary key) needs to be on our schedule eventually.
> I don't know when. The general case is very expensive (lookup position
> and chunk for each position!?), and we are only going to get away with
> it because our bulk-loads for ForcedSource will be spatially-restricted.
>
> -Daniel
>
> On 02/26/2015 12:32 PM, Jacek Becla wrote:
>> So, we said we'd do 10% of DR1. We need to think carefully
>> how we want to look at DR1, because the full data set with
>> indexes, object_extra etc is 1.7 petabyte.
>>
>> I think a fair and realistic test would be to look at 10%
>> of the core data (Object, Source, ForcedSource, Exposures),
>> exercise some scans and joins, but just forget Object_extra
>> (which we can argue will be less frequently used, and testing
>> with it won't really stress qserv software in any serious
>> way anyway. After all, we always ran with Object and Source
>> only in the past too)
>>
>> That basically is ~27 TB + indexes. (The data sizes for DR1:
>>   38 TB Object
>>  186 TB Source
>>   45 TB ForcedSource)
>>
>>
>> We have ~8 TB on each machine at IN2P3, if I recall,
>> so as we said earlier, ~10 machines would be a minimum
>> to run the test, 25 would be comfortable, 50 would be
>> even better.
>>
>> Are we in position to generate Object, Source
>> and ForcedSource tables? Which data set would we be
>> using to start with?
>>
>> Jacek
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1