Hi K-T and Fabrice,

Thanks for the feedback. We'll think more about your feedback as we progress the setup of a multi-node UKIDSS instance in Qserv--and start to experiment with indexing.

I'll let you know when we've made some progress.

Thanks again,
George.

Cc: Teng, Dominique, Fabio, Emmanuel, Qserv Team.

--
Dr George Beckett,
University of Edinburgh

Email: [log in to unmask]
Tel: +44 131 651 3577 (EPCC) /
      +44 131 6688 255 (Astronomy)
Web: http://www.epcc.ed.ac.uk/

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


-----Original Message-----
From: Kian-Tat Lim [mailto:[log in to unmask]] 
Sent: 24 January 2018 22:13
To: Fabrice Jammes <[log in to unmask]>
Cc: BECKETT George <[log in to unmask]>; QSERV-L <[log in to unmask]>; LI Teng <[log in to unmask]>; Emmanuel GANGLER <[log in to unmask]>; Fabio Hernandez <[log in to unmask]>; Dominique Boutigny <[log in to unmask]>
Subject: Re: [QSERV-L] Qserv ingest/ indexing performance, questions about

> In my understanding, we do not index tables on worker for now. 
> [...] and we try not to add indexes on the worker to save space, and time. 

While others can speak to the situation in our current instances, we absolutely must have indexes on the worker nodes to support "small"
queries in full production.  These would include an objectId index (in addition to the "secondary index") and a spatial index (HTM-based in the baseline).

It is true that indexes on measured columns like flux/magnitude are not likely to be worth the space or time given the relatively large fraction of rows that would be returned by a query on the index -- as long as we are using spinning media.  Because of seek penalties, a full table scan is preferred to an index lookup for anything more than tiny numbers of rows.

--
Kian-Tat Lim, LSST Data Management, [log in to unmask]

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1