Jacek,

> Data loading
> ------------
>  - see detailed design (based on discussion Douglas+Jacek) at:
>    https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading
>  - K-T's comment "looks like M/R"
>    - we could leverage hadoop, but would the DM team be
>      happy if we relied on hadoop?

	Why not?  If it meets the need and doesn't cause other problems,
the TCT can approve it.

> Data ingest
> -----------
> Next design discussion: planning to discuss data ingest
>  - Serge, Jacek, Douglas, (maybe others?) sometime next week
>  - we want to propose dividing line between pipelines and ingest
>  - we want to make suggestions such as "ingest = just ingest,
>    don't do astronomy computation, unit conversion, etc as
>    part of ingest, etc...

	Proposal/suggestions accepted; this is pretty much what I've
been advocating -- and Robert has agreed to -- for a while, as
illustrated by IngestSourcesTask (which does do radians to degrees
conversion for angles, but I think that's necessary).  We just haven't
had the time to get people to switch, push this upstream, and kill the
old code yet.

	With this understanding, I don't think there's a lot to ingest
that's different from data loading.

> Documenting failure modes
> -------------------------
>  - except: don't deal with Byzantine failures, these are very
>    expensive to handle. We don't think any other parts
>    of LSST system could survive that kind of failures
>    - so, add a list of "failure modes that we are aware of but
>      we don't plan to handle", and put Byzantine failure there

	I added a comment about Byzantine failures (that you may not
like).

-- 
Kian-Tat Lim, LSST Data Management, [log in to unmask]

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1