Mario

thanks, forwarding to the qserv mailing list

Jacek



On 09/24/2013 02:26 PM, Mario Juric wrote:
> Jacek,
> 	Here's something for you and the team to think about in November: how
> would you modify qserv to download and cache chunks on-demand?
>
> 	Imagine the following scenario: scientist M at university H has access
> to 200+ nodes w. petabytes of storage. They install qserv, and configure
> it in a mode where it initially downloads only the metadata from the
> LSST Archive (e.g., where the chunks are, how many of them are there,
> which tables are where, etc.). When a query is fired against that
> database, it downloads the chunks touched by the query and caches them
> for future use. Note that if no query ever touched a particular table,
> those chunks would never get downloaded (e.g., a stellar astronomer
> would never download the table with bdSamples). After an initial period
> of downloading/caching, the database would asymptote to a steady state
> mode where all frequently queried tables are cached locally (thus
> offloading our database). Bonus points for allowing bootstrapping with
> mass-downloaded data (or sneakernet delivered disks), or expiration of
> rarely accessed chunks.
>
> 	This was a killer feature for LSD [*] (combined with on-the-fly
> cross-matching against remote catalogs, which is a simple extension).
> I'm willing to bet people would use it if we supplied it. It may also
> solve our issue with the number of replicas needed to guard against
> failure, since we could configure our Archive center database to fetch
> any chunks that it doesn't have (e.g., because the nodes have failed)
> from the Chilean or French site.
>
> [*] http://mwscience.net/trac/wiki/ReleaseNotes_0_5_4
>
> Have fun :),
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1