 |
 |
Mario thanks, forwarding to the qserv mailing list Jacek On 09/24/2013 02:26 PM, Mario Juric wrote: > Jacek, > Here's something for you and the team to think about in November: how > would you modify qserv to download and cache chunks on-demand? > > Imagine the following scenario: scientist M at university H has access > to 200+ nodes w. petabytes of storage. They install qserv, and configure > it in a mode where it initially downloads only the metadata from the > LSST Archive (e.g., where the chunks are, how many of them are there, > which tables are where, etc.). When a query is fired against that > database, it downloads the chunks touched by the query and caches them > for future use. Note that if no query ever touched a particular table, > those chunks would never get downloaded (e.g., a stellar astronomer > would never download the table with bdSamples). After an initial period > of downloading/caching, the database would asymptote to a steady state > mode where all frequently queried tables are cached locally (thus > offloading our database). Bonus points for allowing bootstrapping with > mass-downloaded data (or sneakernet delivered disks), or expiration of > rarely accessed chunks. > > This was a killer feature for LSD [*] (combined with on-the-fly > cross-matching against remote catalogs, which is a simple extension). > I'm willing to bet people would use it if we supplied it. It may also > solve our issue with the number of replicas needed to guard against > failure, since we could configure our Archive center database to fetch > any chunks that it doesn't have (e.g., because the nodes have failed) > from the Chilean or French site. > > [*] http://mwscience.net/trac/wiki/ReleaseNotes_0_5_4 > > Have fun :), > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1