Douglas, K-T, Serge, Fabrice, Bill, Jacek
"internal publishing" = providing limited access to a small
selected group that does QA. No science, no publications
based on that data. Data can change
types of updates on internally published data?
- typically just adding quality flags, but can't preclude
larger changes, e,g. might need to fix code and rerun
parts of analysis on full data or subset of data,
output data might change
- so, set reasonable restrictions, document what we
can do/support
hardware used for data loading?
- all dedicated, don't mix with production servers
--> ACTION: need to capture in storage model [Jacek]
non trivial issue that we need to deal with:
one of the nodes when we run partitioning goes down
while we are deleting chunk, we want to continue and
not wait for that node. Later that node comes back up,
need to cleanup data that was supposed to be deleted.
when adding data to existing chunks, use merge
engine, each underlying table can have a different version.
Then combine all underlying table for each merged table,
preferably at the end when all data is QA'ed and ready
to be released for public use.
Will partitioner be sending data to qms?
- yes, eg largest angular separation between source
and object
- it could also produce empty chunk list
watch out for the issue: empty chunk can have non-empty
overlap table. That complicates generating objectId index
partitioner and table prep - are they distributed? Yes!
Feeding data to partitioner - through gpfs
What if we lose cache managed by the TablePrepMgr on one or
a small number of machines?
- are we keeping 2 replicas? (effectively doubling storage)
- or should we rerun and recover using input fits table data?
- probably the latter
data produced by DRP: many complete files, not stream of data
yes, input files will have good spatial locality,
it'd be best if we batch groups of files when loading
So, the plan: DRP keeps producing files and dumping to
gpfs, we consume them in batches, say a new batch every
day or week. Batching helps with segregating writes and
optimizing disk IO
If we reprocess after finding problems during QA before we
make data available to public, we will end up with two versions
of the same objects (positions can change, everything can
change, objectIds stay), so we need to throw existing
chunk corresponding to reprocessed data, and reinsert data
Avoid merging while we are still doing QA, because it might
complicate capturing provenance.
but that means we will have to deal with many files:
20K partitions, so ~1sq deg, so ~few hundreds of input
files per chunk. That is:
20K x ~5 tables x 3 files per table x say 300 merge tables
= almost 100 million files! (distributed, but still it is
a lot)
Create db and empty tables before loading. That is a separate
step. Don't do a "special first load that creates tables"
all data fed to data loader should be in ready-to-load
format, no astronomy related math
Expected schema of the data that data loader gets from DRP?
- same as baseline schema
- but we will need few extra columns, like procHistoryId
column or chunkId. Loader should add these
- loader should also ensure the schema and data match
But data products we are ingesting are used for other
internal things in production, so forcing apps code
to use units we want in database is not a good idea.
We might need a conversion step to realign units. So,
we are proposing:
- write a separate converter that transforms output
from DRP to desired schema
- data loader provides plugin api, apps team
implements the plugin
- this needs to be discuss with the rest of DM
Also, we need to deal with name conversion/mapping
of different fields.
- discuss with the rest of DM
Partitioner should be flexible enough to handle
any schema that it gets, this will be important
for testing/commissioning, etc
Is the schema for chunks always the same, independently
from data quality? Yes!
processing history id will be recorded by orch layer
and stored in provenance tables,
- the proc history id will come with the data files,
- fits metadata is a good place to put it
- we will need to capture additional provenance that
captures how data was loaded, on which machines etc
- create new prochistory id
need to model provenance for loader, or
use task/taskGraph for that
--> ACTION: merge "data ingest" trac page with
"data loading" page, and update the page to
capture what was discussed at this meeting
[Jacek]
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
