 |
 |
Thanks Serge for these clear explanations, In this test I was partitioning SimRefObject but I think it is an error, I'll remove it. (It seems SQL query coverage should be extended, all queries on SimRefObject are in FIXME state ;-)) In my case the dirTable will be Object and the dirColName Source, for the table Source, isn't it? Cheers, Le 18/12/2014 21:33, Serge Monkewitz a écrit : > Well, in testcase 01, SimRefObject and RefSrcMatch are not > partitioned. So Qserv doesn’t really need to know anything about them > for queries to work (at least from the query analysis point of view).. > > If you did decide to partition SimRefObject, it would be its own > director table, and would have dirTable = SimRefObject and dirColName > equal to its PK (simRefObjectId I think it was). > > RefSrcMatch is a match table, which can be used to join between 2 > directors. In this case (I’m not sure where exactly the test case 01 > data is from) it probably links up SimRefObject with the Source table. > So it would have ZK metadata that looks roughly like: > > .../RefSrcMatch/match1 > .../RefSrcMatch/match/dirColName1simRefObjectId > .../RefSrcMatch/match/dirColName2sourceId > .../RefSrcMatch/match/dirTable1SimRefObject > .../RefSrcMatch/match/dirTable2Source > .../RefSrcMatch/match/flagColNameflags > .../RefSrcMatch/partitioning\N > .../RefSrcMatch/partitioning/subChunks0 > > And, since partitioning match table records works differently from > other kinds of records (each record is associated with 2 positions > rather than just 1), you have to invoke sph-partition-matches rather > than sph-partition on it. I don’t know if Andy’s loader supports that yet. > > Cheers, > Serge > > > On Dec 18, 2014, at 6:44 PM, Fabrice Jammes <[log in to unmask] > <mailto:[log in to unmask]>> wrote: > >> -------- Forwarded Message -------- >> Subject: Re: [QSERV-L] LOADER: support for multi-column PK >> Date: Thu, 18 Dec 2014 18:43:24 -0800 >> From: Fabrice Jammes <[log in to unmask]> >> To: Serge Monkewitz <[log in to unmask]> >> >> >> >> Ok, what should be dirTable, dirColName for SimRefObject, indeed it >> doesn't have a link with Object table, but with RefSrcMatch. >> >> Thanks, it helps a lot to understand ;-) >> >> Fabrice >> >> On 12/18/2014 06:33 PM, Serge Monkewitz wrote: >>> Ah. I checked testcase 01, and none of the queries not marked FIXME >>> join between Object and the other tables you mention. In fact, there >>> is only one query that does a join at all (1051_nn.sql >>> <https://dev.lsstcorp.org/cgit/LSST/DMS/testdata/qserv_testdata.git/tree/datasets/case01/queries/1051_nn.sql>), >>> and it joins Object against itself. >>> >>> So if you aren’t loading metadata from .param files anymore and you >>> omit dirTable and dirColName, nothing will break. If you were to run >>> a query that joined say Object and Source, the query analysis code >>> would complain. >>> >>> I didn’t check the second test case. >>> >>> >>> On Dec 18, 2014, at 6:26 PM, Fabrice Jammes <[log in to unmask] >>> <mailto:[log in to unmask]>> wrote: >>> >>>> It seems it is: >>>> >>>> Database changed >>>> mysql> show tables; >>>> +----------------------------------+ >>>> | Tables_in_qservTest_case01_qserv | >>>> +----------------------------------+ >>>> | Filter | >>>> | LeapSeconds | >>>> | Logs | >>>> | Object | >>>> | ObjectFullOverlap_6630 | >>>> | ObjectFullOverlap_6631 | >>>> | ObjectFullOverlap_6800 | >>>> | ObjectFullOverlap_6801 | >>>> | ObjectFullOverlap_6968 | >>>> | ObjectFullOverlap_6970 | >>>> | ObjectFullOverlap_6971 | >>>> | ObjectFullOverlap_7138 | >>>> | ObjectFullOverlap_7140 | >>>> | ObjectFullOverlap_7308 | >>>> | ObjectFullOverlap_7478 | >>>> | ObjectFullOverlap_7648 | >>>> | Object_1234567890 | >>>> | Object_6630 | >>>> | Object_6631 | >>>> | Object_6800 | >>>> | Object_6801 | >>>> | Object_6968 | >>>> | Object_6970 | >>>> | Object_6971 | >>>> | Object_7138 | >>>> | Object_7140 | >>>> | Object_7308 | >>>> | Object_7310 | >>>> | Object_7478 | >>>> | Object_7648 | >>>> | RefSrcMatch | >>>> | Science_Ccd_Exposure | >>>> | SimRefObject | >>>> | Source | >>>> | Source_1234567890 | >>>> | Source_6630 | >>>> | Source_6631 | >>>> | Source_6800 | >>>> | Source_6801 | >>>> | Source_6968 | >>>> | Source_6970 | >>>> | Source_6971 | >>>> | Source_7138 | >>>> | Source_7140 | >>>> | Source_7308 | >>>> | Source_7310 | >>>> | Source_7478 | >>>> | Source_7648 | >>>> | sdqa_Metric | >>>> +----------------------------------+ >>>> 49 rows in set (0,00 sec) >>>> >>>> >>>> On 12/18/2014 06:20 PM, Serge Monkewitz wrote: >>>>> On Dec 18, 2014, at 6:12 PM, Fabrice Jammes >>>>> <[log in to unmask] <mailto:[log in to unmask]>> wrote: >>>>> >>>>>> DeepSource si a view, shouldn't it be RunDeepSource the director >>>>>> table? >>>>> In the Winter2013 release, yes it is. But RunDeepSource contains a >>>>> bunch of extra columns and uses pipeline column naming >>>>> conventions, so I just dumped from the view instead. Note I had to >>>>> do other things to make it work in qserv (IIRC I changed the type >>>>> of BIT columns to TINYINT). Unless the goal is to duplicate the >>>>> plain mysql setup from stripe 82 data as closely as possible, >>>>> let’s not get hung up on these details. >>>>> >>>>>> Furthermore, it seems dirTable and dirColName are optional for >>>>>> now, isn't it? Andy doesn't set it in test case01 and 02 and it >>>>>> works. >>>>> If you don’t specify them though, the query analysis code will not >>>>> let you join between related partitioned tables. Is the data in >>>>> test case01/02 being partitioned at all? >>>>> >>>>> Serge >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the QSERV-L list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>>> >>>> >>> >> >> >> >> >> ------------------------------------------------------------------------ >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >> > > > ------------------------------------------------------------------------ > > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1