LISTSERV 16.5 - QSERV-L Archives

On Jan 17, 2014, at 6:01 AM, Fabrice Jammes <[log in to unmask]> wrote:

On 01/17/2014 02:54 AM, Serge Monkewitz wrote:
Hi Fabrice,

    OK, I read through that page. I just want to point out that you will have to be careful to partition based on the deep source position, not the deep forced source positions. This is because duplicate forced sources are not guaranteed to have identical (ra, decl) coordinates. They will however be associated with the same deep source (have identical deepSourceId values). Thus, to run the deduplication procedure from the trac page on chunks, you'll need to ensure that duplicates always end up in the same chunk.

I’m not sure whether the deep forced source data includes the ra,dec of the deep source it was derived from.
Table RunDeepForcedSource doesn't seem to contains this information.
In this table, the only fields which could contains these spatial informations are :

       cluster_id BIGINT NULL,
       cluster_coord_ra DOUBLE NULL,
       cluster_coord_decl DOUBLE NULL,

Bogdan, Dominique, or Philippe, could you please confirm that the RunDeepForcedSource table doesn't contains the ra,dec of DeepSource entities it references ?

I think those are normally added by source association, and probably aren’t populated here.

If it does not, things will get “interesting”. It should be the case that duplicates have positions that are extremely close to one another. So the first thing to do would be to go ahead and partition with the deep forced source position anyway. To check that no duplicates where split across chunks, you’ll want to set up the partitioner such that each chunk contains exactly one sub-chunk, and such that the overlap radius is non-zero but small (let’s say an arc-minute). This way, the partitioner will split input into chunks, and, for each chunk, provide nearby rows (the overlap). If the two deep forced sources in a duplicate pair are assigned to different chunks, then one will be in the overlap of the chunk for the other, and vice-versa.
ok, very good idea, I fully agree with it, and will rely on it.
One question : could duplicate RunDeepForcedSource entities sets have a cardinality greater than 2 ?

I don’t think so. IIRC the pairs arise from pixel overlap between adjacent coadd “tiles" (I’m not sure what the correct terminology is here) of the stripe 82 coadd; those were laid out in a single row meaning that at most 2 “tiles” cover any particular position (i.e. deep source).

Maybe we could tell scientists that for entities placed near the limit of a chunk, related entities like DeepSource of DeepForcedSource could be placed in the nearest neighbor chunk ?
Furthermore, Osman plans to use MySQL merge engine to manage chunk union, this technology should solve this side effect problem (except at the border of the zone covered by the union of contiguous chunks), shouldn't it ?
This would allow scientist to have consistent data in a significant area (except on the area border) and could be enough for this first test? Don't you think so ?

For a first test I think this is perfectly fine.

If no, I think two new questions are opened :
- couldn't we assign the whole pair to the chunk containing the DeepSource entity referenced by the pair ? I think this would allow spatial queries on DeepSource (ra,dec) to be consistent.

You certainly could. I’m not sure what you mean by consistent spatial queries though - the chunking of RunDeepForcedSource serves only to bring each chunk down to a manageable size and would otherwise be hidden via the MERGE engine. I was thinking that DeepSource would be loaded as a single table, since it is so small.

- in case the user issue a spatial query on DeepForcedSource (ra,dec), he could lost an element of a pair if it has been placed in a contiguous chunk (containing the other element of the pair), is this a problem for scientist ? I don't know.

It may or may not be, however scientists would have to account for this even with an unpartitioned table. If that's a concern, we could measure the maximum angular separation between elements of a pair, and tell scientists to expand their search regions by that amount.

In the future, Qserv may solve these questions as it manages chunk overlaps. Maybe, for now, we could let the merge engine do the job ? It seems this would lead to consistent data everywhere except on the merged chunks union border.

Once you’ve assigned split pairs to chunks, I would discard the overlap tables completely, and use the MERGE engine to combine the chunks. I think this avoids any problems, but I’m not sure what you mean by consistent data here, and perhaps I am missing something. Can you elaborate?