 |
 |
I would like to note that the current system requires the extra raObject and declObject columns in ForcedSource, so that table's size will be proportionally larger than it would be in production. Do we actually have ForcedSource data? I think it would be useful to add some artificial data to one of our test cases (generate 10 randomly generated ForcedSource rows per Object row in case01), so we can actually run that test. The code to load child table rows that lack director positioning (other than the director's primary key) needs to be on our schedule eventually. I don't know when. The general case is very expensive (lookup position and chunk for each position!?), and we are only going to get away with it because our bulk-loads for ForcedSource will be spatially-restricted. -Daniel On 02/26/2015 12:32 PM, Jacek Becla wrote: > So, we said we'd do 10% of DR1. We need to think carefully > how we want to look at DR1, because the full data set with > indexes, object_extra etc is 1.7 petabyte. > > I think a fair and realistic test would be to look at 10% > of the core data (Object, Source, ForcedSource, Exposures), > exercise some scans and joins, but just forget Object_extra > (which we can argue will be less frequently used, and testing > with it won't really stress qserv software in any serious > way anyway. After all, we always ran with Object and Source > only in the past too) > > That basically is ~27 TB + indexes. (The data sizes for DR1: > 38 TB Object > 186 TB Source > 45 TB ForcedSource) > > > We have ~8 TB on each machine at IN2P3, if I recall, > so as we said earlier, ~10 machines would be a minimum > to run the test, 25 would be comfortable, 50 would be > even better. > > Are we in position to generate Object, Source > and ForcedSource tables? Which data set would we be > using to start with? > > Jacek > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1