Great, thanks! Where, specifically, is the data located? I would like to load this data into my dev instance of qserv, using the same partitioning parameters you use. -- Bill On Sep 3, 2013, at 3:01 PM, "Daniel L. Wang" <[log in to unmask]> wrote: > You are welcome to use the pt11 data that I am using on lsst-dev01. > > The old partitioner still works, and its performance is adequate at pt11 scale. > > -Daniel > > On 09/03/2013 11:58 AM, Bill Chickering wrote: >> Thanks, Douglas. Unfortunately, neither 60 stripes, 20 substripes nor 85 stripes, 12 substripes seems to work. Do the contents of ObjectPartitions.csv provide any clues?: >> 100,0,1628411,0.0,9.47368421053,0.0,9.0,0.01667,0.0168777936529 >> 118,1,379476,350.526315789,360.0,0.0,9.0,0.01667,0.0168777936529 >> 80,5,1631301,0.0,9.47368421053,-9.0,0.0,0.01667,0.0168777936529 >> 98,6,373153,350.526315789,360.0,-9.0,0.0,0.01667,0.0168777936529 >> >> Meanwhile, if anyone can direct me to an alternative dataset that is partitioned with known parameters, I might be able to work with that. Otherwise, I am unable to work with the latest versions of qserv that incorporate qms. >> >> -- Bill >> >> >> On Sep 3, 2013, at 8:41 AM, Douglas Smith <[log in to unmask]> wrote: >> >>> On 09/02/2013 05:16 PM, Bill Chickering wrote: >>>> Dear Qserv Team - >>>> >>>> I have yet to resolve my metadata/partitioning issues. I am not able to proceed without at least one of the following: >>>> >>>> 1) Repair my present instance of qserv, which uses pt11 object data from /u1/douglas/testing/data. This is broken because I do not know the correct options to provide qms. Perhaps Douglas can provide this information. >>> Ok, I can't find the exact details on how I made this data, it was a while ago >>> there. But it was either 60 stripes, 20 substripes, or 85 stripes, 12 substripes. >>> Most of the partition testing I've done as been with 85 stripes, 12 substripes. >>> But there was a very early test that I did with 60 stripes and 20 substripes, >>> and that might be this data. >>> >>> You can try those options, and see how that goes. >>> >>>> 2) Alternative partitioned data with known qms options. Daniel, I believe you mentioned you could provide this. >>>> >>>> 3) I could partition data myself. To do this, I believe I need the new partitioner. The Trac pages say the source for this can be found at qserv/admin/dupr, however, I do not see this on the master branch. On which branch, or where else, can I find this? >>>> >>> I was going to say, wait, it is there, and it has been there, hasn't it? >>> But I just looked at the install that I setup for the head of the master >>> from last Thurs., and yeah, I don't see that there. Didn't Serge merge >>> that into master yet? >>> >>> Douglas >>> >>> >>>> Thanks in advance, >>>> Bill >>>> >>>> Begin forwarded message: >>>> >>>>> *From: *Bill Chickering <[log in to unmask] <mailto:[log in to unmask]>> >>>>> *Subject: **Fwd: QMS and partitioning* >>>>> *Date: *September 1, 2013 8:08:36 PM PDT >>>>> *To: *Douglas Smith <[log in to unmask] <mailto:[log in to unmask]>> >>>>> >>>>> Hi Douglas - >>>>> >>>>> Still haven't resolved my metadata issues. Could you please send me the options you use for the pt11 data (i.e. the data at /u1/douglas/testing/data on lsst-dev03). >>>>> >>>>> Thanks, >>>>> Bill >>>>> >>>>> Begin forwarded message: >>>>> >>>>>> *From: *Bill Chickering <[log in to unmask] <mailto:[log in to unmask]>> >>>>>> *Subject: **QMS and partitioning* >>>>>> *Date: *August 28, 2013 7:25:12 PM PDT >>>>>> *To: *Douglas Smith <[log in to unmask] <mailto:[log in to unmask]>> >>>>>> *Cc: *Serge Monkewitz <[log in to unmask] <mailto:[log in to unmask]>> >>>>>> >>>>>> Hi Douglas - >>>>>> >>>>>> (Douglas, I think you're the right person for this email. Serge, if you would be so awesome as to scan through this email, I'd really appreciate it. I believe you wrote the partitioning logic, so perhaps you'll see what I'm doing wrong.) >>>>>> >>>>>> On the surface, my master branch instance of qserv seems to be working correctly. Some issues remain, however. In particular, when I submit the query: "select count(*) from Object;", only records from chunk 80 are included. I have three other chunks, however: 98, 100, and 118. These three chunks are incorrectly excluded from all of my queries. I suspect this is due to incorrect meta data. >>>>>> >>>>>> I'm using the pt11 data from /u1/douglas/testing/data/ on last-dev03. I've only loaded the Object table. Meanwhile, the script I'm using to initialize my QMS data is for pt12 data, as evidenced by its name: qms_setup_pt12.sh. Here are the contents of my version of qms_setup_pt12.sh: >>>>>> cd $SANDBOX >>>>>> export PATH=$SANDBOX/bin:$PATH >>>>>> export PYTHONPATH=$SANDBOX/qserv/meta/dist >>>>>> >>>>>> #reset all meta data: >>>>>> >>>>>> ./qserv/meta/bin/metaClientTool.py destroyMeta >>>>>> ./qserv/meta/bin/metaClientTool.py installMeta >>>>>> >>>>>> #setup database and tables >>>>>> >>>>>> #./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=85 nSubStripes=12 >>>>>> ./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=20 nSubStripes=12 >>>>>> >>>>>> ./qserv/meta/bin/metaClientTool.py createTable LSST tableName=Object partitioning=on schemaFile=$SANDBOX/qserv/master/examples/sampleData/Object.sql overlap=0.025 phiColName=ra_PS thetaColName=decl_PS objIdColName=objectId logicalPart=2 physChunking=0x0021 >>>>>> >>>>>> Notice that I set nStripes=20. Originally, this value was different. Daniel made the observation that within ObjectPartitions.csv, the sixth or seventh columns contained +/-9.0, which implies 20 stripes. But what about nSubStripes and the other options provided to the createDb command? Are erroneous option values here responsible for why 3 of 4 of my chunks are ignored? >>>>>> >>>>>> Any insight insight will be greatly appreciated. >>>>>> >>>>>> Thanks! >>>>>> Bill >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the QSERV-L list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>>> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1