Hi Andy,

On Dec 19, 2014, at 12:09 PM, Salnikov, Andrei A. <[log in to unmask]> wrote:

> we just talked with Fabrice about one of the problems that new 
> data loader created for Fabrice and we need to understand how
> to fix it. The essence of it is that data loader tries to create
> index table in qservMeta for every partitioned table. This breaks 
> if when the table's primary key has more than one colum because 
> data loader only supports one-column PK. 

By index, do you mean the PK -> chunkId secondary index? If so, I think we only plan to provide this for director tables. For a table like Source, or the AvgForcedPhotYearly table Fabrice was trying to load, I don’t think such an index is expected. (Daniel, please correct me if I’m wrong).

> To understand how to fix it I'd like to get an answer to few 
> questions:
> - Do we need an index for every partitioned table? If not then 
>  we should add a parameter to config file which disables index
>  generation for specific tables. Or do we need index only for 
>  director table?

I think the answer is only for directors.

> - Do we need an index for tables which have multi-column PK, will
>  qserv even support this? If yes then index table needs to have 
>  the same PK columns as the original table. If not then I can 
>  just skip generating index for those problematic tables.

For now, I believe we should stick with single column (integer) PKs for directors. Supporting multi-column PKs is of course doable, but it would complicate query analysis. We'd have to look for multiple equality predicate parse tree nodes ANDed together (or worse, split over multiple ON clauses) to identify equijoins.

> Related questions:
> - should duplicator support multi-column PK in the future?
>  I guess 'id' option in config file needs to be a list in this 
>  case.

My 2 cents: please no. The duplicator is intended as a stop-gap way to generate lots of data for testing, and is already complicated enough. If I cannot assume that the director and table PKs are 64 bit integers, then generating unique IDs for duplicated records becomes much harder, and in some cases  impossible.

I also don’t think that we get a lot of value out of the effort that would be needed to do this, but others may disagree.

> - what is the official name for the qservMeta index, I think 
>  "secondary index" is mentioned, but this does not make too
>  much sens to me.

That’s a question for Daniel. I think of (chunkId, subChunkId) as the primary way of looking things up in the Qserv system, so it makes sense to me that an index on director PK be “secondary".

Cheers,
Serge
########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1