 |
 |
Dear Qserv Team - I have yet to resolve my metadata/partitioning issues. I am not able to proceed without at least one of the following: 1) Repair my present instance of qserv, which uses pt11 object data from /u1/douglas/testing/data. This is broken because I do not know the correct options to provide qms. Perhaps Douglas can provide this information. 2) Alternative partitioned data with known qms options. Daniel, I believe you mentioned you could provide this. 3) I could partition data myself. To do this, I believe I need the new partitioner. The Trac pages say the source for this can be found at qserv/admin/dupr, however, I do not see this on the master branch. On which branch, or where else, can I find this? Thanks in advance, Bill Begin forwarded message: > From: Bill Chickering <[log in to unmask]> > Subject: Fwd: QMS and partitioning > Date: September 1, 2013 8:08:36 PM PDT > To: Douglas Smith <[log in to unmask]> > > Hi Douglas - > > Still haven't resolved my metadata issues. Could you please send me the options you use for the pt11 data (i.e. the data at /u1/douglas/testing/data on lsst-dev03). > > Thanks, > Bill > > Begin forwarded message: > >> From: Bill Chickering <[log in to unmask]> >> Subject: QMS and partitioning >> Date: August 28, 2013 7:25:12 PM PDT >> To: Douglas Smith <[log in to unmask]> >> Cc: Serge Monkewitz <[log in to unmask]> >> >> Hi Douglas - >> >> (Douglas, I think you're the right person for this email. Serge, if you would be so awesome as to scan through this email, I'd really appreciate it. I believe you wrote the partitioning logic, so perhaps you'll see what I'm doing wrong.) >> >> On the surface, my master branch instance of qserv seems to be working correctly. Some issues remain, however. In particular, when I submit the query: "select count(*) from Object;", only records from chunk 80 are included. I have three other chunks, however: 98, 100, and 118. These three chunks are incorrectly excluded from all of my queries. I suspect this is due to incorrect meta data. >> >> I'm using the pt11 data from /u1/douglas/testing/data/ on last-dev03. I've only loaded the Object table. Meanwhile, the script I'm using to initialize my QMS data is for pt12 data, as evidenced by its name: qms_setup_pt12.sh. Here are the contents of my version of qms_setup_pt12.sh: >> cd $SANDBOX >> export PATH=$SANDBOX/bin:$PATH >> export PYTHONPATH=$SANDBOX/qserv/meta/dist >> >> #reset all meta data: >> >> ./qserv/meta/bin/metaClientTool.py destroyMeta >> ./qserv/meta/bin/metaClientTool.py installMeta >> >> #setup database and tables >> >> #./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=85 nSubStripes=12 >> ./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=20 nSubStripes=12 >> >> ./qserv/meta/bin/metaClientTool.py createTable LSST tableName=Object partitioning=on schemaFile=$SANDBOX/qserv/master/examples/sampleData/Object.sql overlap=0.025 phiColName=ra_PS thetaColName=decl_PS objIdColName=objectId logicalPart=2 physChunking=0x0021 >> >> Notice that I set nStripes=20. Originally, this value was different. Daniel made the observation that within ObjectPartitions.csv, the sixth or seventh columns contained +/-9.0, which implies 20 stripes. But what about nSubStripes and the other options provided to the createDb command? Are erroneous option values here responsible for why 3 of 4 of my chunks are ignored? >> >> Any insight insight will be greatly appreciated. >> >> Thanks! >> Bill > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1