Hi Jacek,

On 09/13/2013 12:15 PM, Becla, Jacek wrote:
> ##### large area (500 sq deg) object scan with simple aggregation
> SELECT  AVG(parallax)
> FROM    Object
> WHERE   qserv_areaspec_box(-3,7,10, 60)
This works:

SELECT  AVG(gFlux_PS)
FROM    Object
WHERE   qserv_areaspec_box(-3,10,7, 60)

I fixed the coordinate ordering to something you probably meant. We can 
argue about the ordering of coordinates in qserv_areaspec_box() later.

> ##### full object scan, density per declination zone
> SELECT  ROUND(decl_PS, 0) AS declRounded, COUNT(*) AS theCount
> FROM     Object
> GROUP BY declRounded
> ORDER BY theCount;
Now it works.

> ##### near neighbor, 100 sqdeg
> SELECT   COUNT(*)
> FROM     Object o1, Object o2
> WHERE    qserv_areaspec_box(-5,-5, 5, 5)
>     AND    qserv_angSep(o1.ra_PS, o1.decl_PS, o2.ra_PS, o2.decl_PS) < 0.1;
This is really slow, primarily because so much work is wasted on 
creating/destroying empty subchunks. The master doesn't (though it 
could) know what subchunks are empty, so it (a) generates queries acting 
on empty subchunks and (b) requests those empty subchunks to be created.

Solutions:
1. Move subchunk query generation to the worker (protocol changes, 
dependency shuffling to promote the query templating code into common). 
This doesn't really fix anything by itself, except it improves dispatch 
speed.

2. Let the master know what subChunks are populated. We can build an 
index pretty easily from the objectid index. Now the master can skip 
worrying about empty subchunks. (similarly, we can create the 
emptychunks file). Everyone wins here, minus the time to create the 
index. I think we can create it in-memory on startup and maybe cache it 
on disk to avoid reading on startup.

3. Have the worker figure out what subchunks are populated. The worker 
is really in the right position to do this. Coupled with 1., it would be 
comparable to 2.

We can probably do 2. It's the easiest. But not right now, not today.

> plus if maybe you could get that to work...
>
> SELECT
>     ROUND(scisql_fluxToAbMag(uFlux_PS)-scisql_fluxToAbMag(gFlux_PS), 0)
> AS UG,
>     ROUND(scisql_fluxToAbMag(gFlux_PS)-scisql_fluxToAbMag(rFlux_PS), 0)
> AS GR,
>     ROUND(scisql_fluxToAbMag(rFlux_PS)-scisql_fluxToAbMag(iFlux_PS), 0)
> AS RI,
>     ROUND(scisql_fluxToAbMag(iFlux_PS)-scisql_fluxToAbMag(zFlux_PS), 0)
> AS IZ,
>     ROUND(scisql_fluxToAbMag(zFlux_PS)-scisql_fluxToAbMag(yFlux_PS), 0) AS ZY
> FROM Object
> WHERE
>     scisql_fluxToAbMag(uFlux_PS)-scisql_fluxToAbMag(gFlux_PS) +
>     scisql_fluxToAbMag(gFlux_PS)-scisql_fluxToAbMag(rFlux_PS) +
>     scisql_fluxToAbMag(rFlux_PS)-scisql_fluxToAbMag(iFlux_PS) +
>     scisql_fluxToAbMag(iFlux_PS)-scisql_fluxToAbMag(zFlux_PS) +
>     scisql_fluxToAbMag(zFlux_PS)-scisql_fluxToAbMag(yFlux_PS) < <x>;
Actually, this might work. It returns no rows with <x> = 1. It probably 
helped that I cleaned up some of the alias handling for aggregation. 
Don't have time to test it right now.

code on u/danielw/sScan5

-Daniel

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1