Jacek,
> Data loading
> ------------
> - see detailed design (based on discussion Douglas+Jacek) at:
> https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading
> - K-T's comment "looks like M/R"
> - we could leverage hadoop, but would the DM team be
> happy if we relied on hadoop?
Why not? If it meets the need and doesn't cause other problems,
the TCT can approve it.
> Data ingest
> -----------
> Next design discussion: planning to discuss data ingest
> - Serge, Jacek, Douglas, (maybe others?) sometime next week
> - we want to propose dividing line between pipelines and ingest
> - we want to make suggestions such as "ingest = just ingest,
> don't do astronomy computation, unit conversion, etc as
> part of ingest, etc...
Proposal/suggestions accepted; this is pretty much what I've
been advocating -- and Robert has agreed to -- for a while, as
illustrated by IngestSourcesTask (which does do radians to degrees
conversion for angles, but I think that's necessary). We just haven't
had the time to get people to switch, push this upstream, and kill the
old code yet.
With this understanding, I don't think there's a lot to ingest
that's different from data loading.
> Documenting failure modes
> -------------------------
> - except: don't deal with Byzantine failures, these are very
> expensive to handle. We don't think any other parts
> of LSST system could survive that kind of failures
> - so, add a list of "failure modes that we are aware of but
> we don't plan to handle", and put Byzantine failure there
I added a comment about Byzantine failures (that you may not
like).
--
Kian-Tat Lim, LSST Data Management, [log in to unmask]
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|