 |
 |
Serge Yes, for sure. I added a section about it to DataLoading page: https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading#Notesonusingmergeengineaspartofdataloading we would end up with 7.2 billion files... Jacek On 10/29/2013 05:39 PM, Serge Monkewitz wrote: > Jacek, > > Tying this in with the proposed use of MERGE for incremental loading, it seems one would have to multiply the number of files by the average number of input files contributing to each sub-chunk. I worry that this takes MERGE over the edge. > > Serge > >> On Oct 29, 2013, at 16:31, Jacek Becla <[log in to unmask]> wrote: >> >> K-T, >> >>> I don't understand this. I thought the entire point was to use >>> the individual subchunk tables for things like near neighbor, not the >>> merged table, which would be used for all other types of queries. In >>> that mode, the performance doesn't look too bad: >> >> Yes, indeed, but I and Daniel thought we'd check "just in case", >> hoping that perhaps merge engine is smarter than we think it is, >> which would simplify our design a little. >> >>> The open files may or may not be a problem; having tens of >>> thousands open is usually not an issue. >> >> That can be easily tested. >> >> We will keep the merge engine as an option. I like its simplicity... >> >> Jacek >> >> _______________________________________________ >> LSST-data mailing list >> [log in to unmask] >> http://listserv.lsstcorp.org/mailman/listinfo/lsst-data ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1