 |
 |
You are welcome to use the pt11 data that I am using on lsst-dev01. The old partitioner still works, and its performance is adequate at pt11 scale. -Daniel On 09/03/2013 11:58 AM, Bill Chickering wrote: > Thanks, Douglas. Unfortunately, neither 60 stripes, 20 substripes nor 85 stripes, 12 substripes seems to work. Do the contents of ObjectPartitions.csv provide any clues?: > 100,0,1628411,0.0,9.47368421053,0.0,9.0,0.01667,0.0168777936529 > 118,1,379476,350.526315789,360.0,0.0,9.0,0.01667,0.0168777936529 > 80,5,1631301,0.0,9.47368421053,-9.0,0.0,0.01667,0.0168777936529 > 98,6,373153,350.526315789,360.0,-9.0,0.0,0.01667,0.0168777936529 > > Meanwhile, if anyone can direct me to an alternative dataset that is partitioned with known parameters, I might be able to work with that. Otherwise, I am unable to work with the latest versions of qserv that incorporate qms. > > -- Bill > > > On Sep 3, 2013, at 8:41 AM, Douglas Smith <[log in to unmask]> wrote: > >> On 09/02/2013 05:16 PM, Bill Chickering wrote: >>> Dear Qserv Team - >>> >>> I have yet to resolve my metadata/partitioning issues. I am not able to proceed without at least one of the following: >>> >>> 1) Repair my present instance of qserv, which uses pt11 object data from /u1/douglas/testing/data. This is broken because I do not know the correct options to provide qms. Perhaps Douglas can provide this information. >> Ok, I can't find the exact details on how I made this data, it was a while ago >> there. But it was either 60 stripes, 20 substripes, or 85 stripes, 12 substripes. >> Most of the partition testing I've done as been with 85 stripes, 12 substripes. >> But there was a very early test that I did with 60 stripes and 20 substripes, >> and that might be this data. >> >> You can try those options, and see how that goes. >> >>> 2) Alternative partitioned data with known qms options. Daniel, I believe you mentioned you could provide this. >>> >>> 3) I could partition data myself. To do this, I believe I need the new partitioner. The Trac pages say the source for this can be found at qserv/admin/dupr, however, I do not see this on the master branch. On which branch, or where else, can I find this? >>> >> I was going to say, wait, it is there, and it has been there, hasn't it? >> But I just looked at the install that I setup for the head of the master >> from last Thurs., and yeah, I don't see that there. Didn't Serge merge >> that into master yet? >> >> Douglas >> >> >>> Thanks in advance, >>> Bill >>> >>> Begin forwarded message: >>> >>>> *From: *Bill Chickering <[log in to unmask] <mailto:[log in to unmask]>> >>>> *Subject: **Fwd: QMS and partitioning* >>>> *Date: *September 1, 2013 8:08:36 PM PDT >>>> *To: *Douglas Smith <[log in to unmask] <mailto:[log in to unmask]>> >>>> >>>> Hi Douglas - >>>> >>>> Still haven't resolved my metadata issues. Could you please send me the options you use for the pt11 data (i.e. the data at /u1/douglas/testing/data on lsst-dev03). >>>> >>>> Thanks, >>>> Bill >>>> >>>> Begin forwarded message: >>>> >>>>> *From: *Bill Chickering <[log in to unmask] <mailto:[log in to unmask]>> >>>>> *Subject: **QMS and partitioning* >>>>> *Date: *August 28, 2013 7:25:12 PM PDT >>>>> *To: *Douglas Smith <[log in to unmask] <mailto:[log in to unmask]>> >>>>> *Cc: *Serge Monkewitz <[log in to unmask] <mailto:[log in to unmask]>> >>>>> >>>>> Hi Douglas - >>>>> >>>>> (Douglas, I think you're the right person for this email. Serge, if you would be so awesome as to scan through this email, I'd really appreciate it. I believe you wrote the partitioning logic, so perhaps you'll see what I'm doing wrong.) >>>>> >>>>> On the surface, my master branch instance of qserv seems to be working correctly. Some issues remain, however. In particular, when I submit the query: "select count(*) from Object;", only records from chunk 80 are included. I have three other chunks, however: 98, 100, and 118. These three chunks are incorrectly excluded from all of my queries. I suspect this is due to incorrect meta data. >>>>> >>>>> I'm using the pt11 data from /u1/douglas/testing/data/ on last-dev03. I've only loaded the Object table. Meanwhile, the script I'm using to initialize my QMS data is for pt12 data, as evidenced by its name: qms_setup_pt12.sh. Here are the contents of my version of qms_setup_pt12.sh: >>>>> cd $SANDBOX >>>>> export PATH=$SANDBOX/bin:$PATH >>>>> export PYTHONPATH=$SANDBOX/qserv/meta/dist >>>>> >>>>> #reset all meta data: >>>>> >>>>> ./qserv/meta/bin/metaClientTool.py destroyMeta >>>>> ./qserv/meta/bin/metaClientTool.py installMeta >>>>> >>>>> #setup database and tables >>>>> >>>>> #./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=85 nSubStripes=12 >>>>> ./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=20 nSubStripes=12 >>>>> >>>>> ./qserv/meta/bin/metaClientTool.py createTable LSST tableName=Object partitioning=on schemaFile=$SANDBOX/qserv/master/examples/sampleData/Object.sql overlap=0.025 phiColName=ra_PS thetaColName=decl_PS objIdColName=objectId logicalPart=2 physChunking=0x0021 >>>>> >>>>> Notice that I set nStripes=20. Originally, this value was different. Daniel made the observation that within ObjectPartitions.csv, the sixth or seventh columns contained +/-9.0, which implies 20 stripes. But what about nSubStripes and the other options provided to the createDb command? Are erroneous option values here responsible for why 3 of 4 of my chunks are ignored? >>>>> >>>>> Any insight insight will be greatly appreciated. >>>>> >>>>> Thanks! >>>>> Bill >>> >>> ------------------------------------------------------------------------ >>> >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the QSERV-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>> > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1