I think the loader should have some hook for pushing its chunk info.
This can be as simple as some well-defined part of the loader code where
we can switch to invoking some other function that consumes the info
(later, this function will push the info to update the appropriate db).
Some chunk info can be recovered by "polling" (via an API that doesn't
exist but could) xrootd, but not all of it. Think about these 2
questions: (a)Where is chunk X? (b) Is chunk X part of the data set?
Xrootd can help us answer (a), but not (b), which is a property of the
data. If the secondary index ("objectId index") is available and
correct, it can answer (b)--but qserv will make use of the info for (b)
while building that index.
-Daniel
On 11/26/2014 02:17 PM, Jacek Becla wrote:
> AndyS and I just talked about it... his loader code has info about
> chunks, and it'd be useful to save it. Idea: keep info about chunks
> (and in the future about replicas) together with runtime query
> metadata (in mysql). Recover from failures by polling xrootd.
>
> Tentative plan for short term: implement function that provides
> info about chunks. Later, use the function to get that info and
> store it in runtime metadata system.
>
> Thoughts?
>
> p.s. maybe we could ask Fritz to implement runtime metadata (for
> queries, and chunks)
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
