 |
 |
Ok, for now, I'll use -i " " options and a 'director" entry in python hard-coded integration test configuration. Integration test configuration is hard-coded in python but it moves a lot (for example new loader remove 70% of the parameters and the new director parameter will disappear with Andy ticket) so embedding it in configuration files would add lots of extra job for a limited added-value. Nevetheless I open to better solutions. Any objection, or (even better ;-) ) suggestions about this? Cheers On 12/19/2014 12:59 PM, Salnikov, Andrei A. wrote: > Thanks everybody for answering my questions. I'll restrict > index generation to director table only then, will make new > ticket for that. > > Cheers, > Andy > > > Becla, Jacek wrote on 2014-12-19: >> Yes, the plan always was to build a "secondary index" >> (basically a mapping from objectId to chunkId and subChunkId) >> *only for the Object table* (or, the director table, speaking >> in general terms). >> >> It is perfectly fine to restrict secondary index to >> a single column, but we should >> a) document it, >> b) catch when someone tries multi-column, and gracefully fail >> >> Jacek >> >> >> >> On 12/19/2014 12:34 PM, Serge Monkewitz wrote: >>> Hi Andy, >>> >>> On Dec 19, 2014, at 12:09 PM, Salnikov, Andrei A. >>> <[log in to unmask]> wrote: >>> >>>> we just talked with Fabrice about one of the problems that new >>>> data loader created for Fabrice and we need to understand how >>>> to fix it. The essence of it is that data loader tries to create >>>> index table in qservMeta for every partitioned table. This breaks >>>> if when the table's primary key has more than one colum because >>>> data loader only supports one-column PK. >>> By index, do you mean the PK -> chunkId secondary index? If so, I think >> we only plan to provide this for director tables. For a table like Source, >> or the AvgForcedPhotYearly table Fabrice was trying to load, I don't think >> such an index is expected. (Daniel, please correct me if I'm wrong). >>>> To understand how to fix it I'd like to get an answer to few >>>> questions: >>>> - Do we need an index for every partitioned table? If not then >>>> we should add a parameter to config file which disables index >>>> generation for specific tables. Or do we need index only for >>>> director table? >>> I think the answer is only for directors. >>> >>>> - Do we need an index for tables which have multi-column PK, will >>>> qserv even support this? If yes then index table needs to have >>>> the same PK columns as the original table. If not then I can >>>> just skip generating index for those problematic tables. >>> For now, I believe we should stick with single column (integer) PKs for >> directors. Supporting multi-column PKs is of course doable, but it would >> complicate query analysis. We'd have to look for multiple equality >> predicate parse tree nodes ANDed together (or worse, split over multiple >> ON clauses) to identify equijoins. >>>> Related questions: >>>> - should duplicator support multi-column PK in the future? >>>> I guess 'id' option in config file needs to be a list in this >>>> case. >>> My 2 cents: please no. The duplicator is intended as a stop-gap way to >> generate lots of data for testing, and is already complicated enough. If I >> cannot assume that the director and table PKs are 64 bit integers, then >> generating unique IDs for duplicated records becomes much harder, and in >> some cases impossible. >>> I also don't think that we get a lot of value out of the effort that >>> would be needed to do this, but others may disagree. >>> >>>> - what is the official name for the qservMeta index, I think >>>> "secondary index" is mentioned, but this does not make too >>>> much sens to me. >>> That's a question for Daniel. I think of (chunkId, subChunkId) as the >> primary way of looking things up in the Qserv system, so it makes sense to >> me that an index on director PK be "secondary". >>> Cheers, >>> Serge >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the QSERV-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>> > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1