 |
 |
I added an epic https://jira.lsstcorp.org/browse/DM-2187 to capture loading tables with no ra/decl Jacek On 02/26/2015 12:39 PM, Daniel L. Wang wrote: > I would like to note that the current system requires the extra raObject > and declObject columns in ForcedSource, so that table's size will be > proportionally larger than it would be in production. Do we actually > have ForcedSource data? I think it would be useful to add some > artificial data to one of our test cases (generate 10 randomly generated > ForcedSource rows per Object row in case01), so we can actually run that > test. > > The code to load child table rows that lack director positioning (other > than the director's primary key) needs to be on our schedule eventually. > I don't know when. The general case is very expensive (lookup position > and chunk for each position!?), and we are only going to get away with > it because our bulk-loads for ForcedSource will be spatially-restricted. > > -Daniel > > On 02/26/2015 12:32 PM, Jacek Becla wrote: >> So, we said we'd do 10% of DR1. We need to think carefully >> how we want to look at DR1, because the full data set with >> indexes, object_extra etc is 1.7 petabyte. >> >> I think a fair and realistic test would be to look at 10% >> of the core data (Object, Source, ForcedSource, Exposures), >> exercise some scans and joins, but just forget Object_extra >> (which we can argue will be less frequently used, and testing >> with it won't really stress qserv software in any serious >> way anyway. After all, we always ran with Object and Source >> only in the past too) >> >> That basically is ~27 TB + indexes. (The data sizes for DR1: >> 38 TB Object >> 186 TB Source >> 45 TB ForcedSource) >> >> >> We have ~8 TB on each machine at IN2P3, if I recall, >> so as we said earlier, ~10 machines would be a minimum >> to run the test, 25 would be comfortable, 50 would be >> even better. >> >> Are we in position to generate Object, Source >> and ForcedSource tables? Which data set would we be >> using to start with? >> >> Jacek >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1