 |
 |
sph-partition-matches is not supported yet. I think we need to document better two things: - CSS keys/values which are used by qserv - all possible configuration parameters used by partitioner and data loader Cheers, Andy From: [log in to unmask] [mailto:[log in to unmask]] On Behalf Of Serge Monkewitz Sent: Thursday, December 18, 2014 9:33 PM To: Fabrice Jammes Cc: qserv-l Subject: Re: [QSERV-L] Fwd: Re: [QSERV-L] LOADER: support for multi-column PK Well, in testcase 01, SimRefObject and RefSrcMatch are not partitioned. So Qserv doesn't really need to know anything about them for queries to work (at least from the query analysis point of view).. If you did decide to partition SimRefObject, it would be its own director table, and would have dirTable = SimRefObject and dirColName equal to its PK (simRefObjectId I think it was). RefSrcMatch is a match table, which can be used to join between 2 directors. In this case (I'm not sure where exactly the test case 01 data is from) it probably links up SimRefObject with the Source table. So it would have ZK metadata that looks roughly like: .../RefSrcMatch/match 1 .../RefSrcMatch/match/dirColName1 simRefObjectId .../RefSrcMatch/match/dirColName2 sourceId .../RefSrcMatch/match/dirTable1 SimRefObject .../RefSrcMatch/match/dirTable2 Source .../RefSrcMatch/match/flagColName flags .../RefSrcMatch/partitioning \N .../RefSrcMatch/partitioning/subChunks 0 And, since partitioning match table records works differently from other kinds of records (each record is associated with 2 positions rather than just 1), you have to invoke sph-partition-matches rather than sph-partition on it. I don't know if Andy's loader supports that yet. Cheers, Serge On Dec 18, 2014, at 6:44 PM, Fabrice Jammes <[log in to unmask]<mailto:[log in to unmask]>> wrote: -------- Forwarded Message -------- Subject: Re: [QSERV-L] LOADER: support for multi-column PK Date: Thu, 18 Dec 2014 18:43:24 -0800 From: Fabrice Jammes <[log in to unmask]><mailto:[log in to unmask]> To: Serge Monkewitz <[log in to unmask]><mailto:[log in to unmask]> Ok, what should be dirTable, dirColName for SimRefObject, indeed it doesn't have a link with Object table, but with RefSrcMatch. Thanks, it helps a lot to understand ;-) Fabrice On 12/18/2014 06:33 PM, Serge Monkewitz wrote: Ah. I checked testcase 01, and none of the queries not marked FIXME join between Object and the other tables you mention. In fact, there is only one query that does a join at all (1051_nn.sql<https://dev.lsstcorp.org/cgit/LSST/DMS/testdata/qserv_testdata.git/tree/datasets/case01/queries/1051_nn.sql>), and it joins Object against itself. So if you aren't loading metadata from .param files anymore and you omit dirTable and dirColName, nothing will break. If you were to run a query that joined say Object and Source, the query analysis code would complain. I didn't check the second test case. On Dec 18, 2014, at 6:26 PM, Fabrice Jammes <[log in to unmask]<mailto:[log in to unmask]>> wrote: It seems it is: Database changed mysql> show tables; +----------------------------------+ | Tables_in_qservTest_case01_qserv | +----------------------------------+ | Filter | | LeapSeconds | | Logs | | Object | | ObjectFullOverlap_6630 | | ObjectFullOverlap_6631 | | ObjectFullOverlap_6800 | | ObjectFullOverlap_6801 | | ObjectFullOverlap_6968 | | ObjectFullOverlap_6970 | | ObjectFullOverlap_6971 | | ObjectFullOverlap_7138 | | ObjectFullOverlap_7140 | | ObjectFullOverlap_7308 | | ObjectFullOverlap_7478 | | ObjectFullOverlap_7648 | | Object_1234567890 | | Object_6630 | | Object_6631 | | Object_6800 | | Object_6801 | | Object_6968 | | Object_6970 | | Object_6971 | | Object_7138 | | Object_7140 | | Object_7308 | | Object_7310 | | Object_7478 | | Object_7648 | | RefSrcMatch | | Science_Ccd_Exposure | | SimRefObject | | Source | | Source_1234567890 | | Source_6630 | | Source_6631 | | Source_6800 | | Source_6801 | | Source_6968 | | Source_6970 | | Source_6971 | | Source_7138 | | Source_7140 | | Source_7308 | | Source_7310 | | Source_7478 | | Source_7648 | | sdqa_Metric | +----------------------------------+ 49 rows in set (0,00 sec) On 12/18/2014 06:20 PM, Serge Monkewitz wrote: On Dec 18, 2014, at 6:12 PM, Fabrice Jammes <[log in to unmask]<mailto:[log in to unmask]>> wrote: DeepSource si a view, shouldn't it be RunDeepSource the director table? In the Winter2013 release, yes it is. But RunDeepSource contains a bunch of extra columns and uses pipeline column naming conventions, so I just dumped from the view instead. Note I had to do other things to make it work in qserv (IIRC I changed the type of BIT columns to TINYINT). Unless the goal is to duplicate the plain mysql setup from stripe 82 data as closely as possible, let's not get hung up on these details. Furthermore, it seems dirTable and dirColName are optional for now, isn't it? Andy doesn't set it in test case01 and 02 and it works. If you don't specify them though, the query analysis code will not let you join between related partitioned tables. Is the data in test case01/02 being partitioned at all? Serge ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ________________________________ Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ________________________________ Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1