Print

Print
Dear Qserv Team -

I have yet to resolve my metadata/partitioning issues. I am not able to proceed without at least one of the following:

1) Repair my present instance of qserv, which uses pt11 object data from /u1/douglas/testing/data. This is broken because I do not know the correct options to provide qms. Perhaps Douglas can provide this information.

2) Alternative partitioned data with known qms options. Daniel, I believe you mentioned you could provide this.

3) I could partition data myself. To do this, I believe I need the new partitioner. The Trac pages say the source for this can be found at qserv/admin/dupr, however, I do not see this on the master branch. On which branch, or where else, can I find this?

Thanks in advance,
Bill

Begin forwarded message:

From: Bill Chickering <[log in to unmask]>
Subject: Fwd: QMS and partitioning
Date: September 1, 2013 8:08:36 PM PDT
To: Douglas Smith <[log in to unmask]>

Hi Douglas -

Still haven't resolved my metadata issues. Could you please send me the options you use for the pt11 data (i.e. the data at /u1/douglas/testing/data on lsst-dev03).

Thanks,
Bill

Begin forwarded message:

From: Bill Chickering <[log in to unmask]>
Subject: QMS and partitioning
Date: August 28, 2013 7:25:12 PM PDT
To: Douglas Smith <[log in to unmask]>
Cc: Serge Monkewitz <[log in to unmask]>

Hi Douglas -

(Douglas, I think you're the right person for this email. Serge, if you would be so awesome as to scan through this email, I'd really appreciate it. I believe you wrote the partitioning logic, so perhaps you'll see what I'm doing wrong.)

On the surface, my master branch instance of qserv seems to be working correctly. Some issues remain, however. In particular, when I submit the query: "select count(*) from Object;", only records from chunk 80 are included. I have three other chunks, however: 98, 100, and 118. These three chunks are incorrectly excluded from all of my queries. I suspect this is due to incorrect meta data. 

I'm using the pt11 data from /u1/douglas/testing/data/ on last-dev03. I've only loaded the Object table. Meanwhile, the script I'm using to initialize my QMS data is for pt12 data, as evidenced by its name: qms_setup_pt12.sh. Here are the contents of my version of qms_setup_pt12.sh:
cd $SANDBOX
export PATH=$SANDBOX/bin:$PATH
export PYTHONPATH=$SANDBOX/qserv/meta/dist

#reset all meta data:

./qserv/meta/bin/metaClientTool.py destroyMeta
./qserv/meta/bin/metaClientTool.py installMeta

#setup database and tables

#./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=85 nSubStripes=12
./qserv/meta/bin/metaClientTool.py createDb LSST partitioning=on partitioningStrategy=sphBox defaultOverlap_fuzziness=0 defaultOverlap_nearNeighbor=0.025 nStripes=20 nSubStripes=12

./qserv/meta/bin/metaClientTool.py createTable LSST tableName=Object partitioning=on schemaFile=$SANDBOX/qserv/master/examples/sampleData/Object.sql overlap=0.025 phiColName=ra_PS thetaColName=decl_PS objIdColName=objectId logicalPart=2 physChunking=0x0021

Notice that I set nStripes=20. Originally, this value was different. Daniel made the observation that within ObjectPartitions.csv, the sixth or seventh columns contained +/-9.0, which implies 20 stripes. But what about nSubStripes and the other options provided to the createDb command? Are erroneous option values here responsible for why 3 of 4 of my chunks are ignored?

Any insight insight will be greatly appreciated.

Thanks!
Bill



Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1