Hi Daniel,
my current approach is that loader will be a regular python module
(actually a class). You can import and load data and after loading
the list of chunks will be available to client code, one can do
whatever is reasonable with that info.
Cheers,
Andy
Daniel L. Wang wrote on 2014-12-01:
> I think the loader should have some hook for pushing its chunk info.
> This can be as simple as some well-defined part of the loader code where
> we can switch to invoking some other function that consumes the info
> (later, this function will push the info to update the appropriate db).
>
> Some chunk info can be recovered by "polling" (via an API that doesn't
> exist but could) xrootd, but not all of it. Think about these 2
> questions: (a)Where is chunk X? (b) Is chunk X part of the data set?
> Xrootd can help us answer (a), but not (b), which is a property of the
> data. If the secondary index ("objectId index") is available and
> correct, it can answer (b)--but qserv will make use of the info for (b)
> while building that index.
>
> -Daniel
>
>
>
> On 11/26/2014 02:17 PM, Jacek Becla wrote:
>> AndyS and I just talked about it... his loader code has info about
>> chunks, and it'd be useful to save it. Idea: keep info about chunks
>> (and in the future about replicas) together with runtime query
>> metadata (in mysql). Recover from failures by polling xrootd.
>>
>> Tentative plan for short term: implement function that provides
>> info about chunks. Later, use the function to get that info and
>> store it in runtime metadata system.
>>
>> Thoughts?
>>
>> p.s. maybe we could ask Fritz to implement runtime metadata (for
>> queries, and chunks)
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
