 |
 |
Hi Andy, Serge and Vaikunth, The previous duplicator (partition.py) was running on partitioned data (I don't know how it was doing to not duplicate PK). Whereas I think that sph-duplicate is running on non-partitioned data, could you confirm it to us please Serge? sph-duplicate --help provide good documentation on options, but I don't see how to provide an input data to this tool? If I'm right, Serge, could you please add it? Thanks, Fabrice On 12/19/2014 09:54 AM, Salnikov, Andrei A. wrote: > > Hi Vaikunth, > > I think it's better to add duplicator call to data-loader itself. It's > probably > > the same amount of work if you do it in integration scripts but > duplication should > > happen after partitioning so logically we should just stick into a > loader script > > after call to partitioner. I think it will be straightforward to do, > there should > > not be much interaction with other code (except passing all required > parameters). > > And it can probably be done in DM-1570 if you do not want to open new > ticket. > > Cheers, > Andy > > *From:*Vaikunth Thukral [mailto:[log in to unmask]] > *Sent:* Friday, December 19, 2014 9:23 AM > *To:* Salnikov, Andrei A. > *Cc:* qserv-l > *Subject:* Re: [QSERV-L] Adding Duplicator to Integration Tests > > Hi Andy, > > I saw from the code that the loader also enables loading of > pre-partitioned data, something that the duplicator produces. Another > possibility could be to use the duplicator upstream and use the loader > as-is on this now-partitioned data. This could be a quicker solution > for adding the new test case, but if the ideal way is to just have the > loader do the duplication then I won't mind looking into it. Would > that technically block this issue (1570) then? What do you guys think? > > -Vaikunth > > On Fri, Dec 19, 2014 at 12:38 AM, Salnikov, Andrei A. > <[log in to unmask] <mailto:[log in to unmask]>> wrote: > > Hi Vaikunth, > > duplication should be a part of data loading and needs to be added to > data-loader > > script eventually. Do you want to work on that? > > > Cheers, > Andy > > *From:*[log in to unmask] > <mailto:[log in to unmask]>[mailto:[log in to unmask] > <mailto:[log in to unmask]>] *On Behalf Of *Vaikunth Thukral > *Sent:* Thursday, December 18, 2014 9:14 PM > *To:* qserv-l > *Subject:* [QSERV-L] Adding Duplicator to Integration Tests > > Hi all, > > Serge, Fabrice and I have been talking about how to go forward with > adding the test case of using the duplicator (DM-1570). I know Fabrice > is working on using AndyS's new loader to load data for integration > tests using partitioning. > > Would it be acceptable to modify the loader script to use/add a > duplication option instead of just partitioning? > > -Vaikunth > > ------------------------------------------------------------------------ > > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > > > ------------------------------------------------------------------------ > > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1