Hello Serge,
It seems the partitioning process runned successfully, and Osman
Aidel, our CC-IN2P3 database expert, is ready to start loading it
with MariaDB.
Nevertheless the produced data size is slighty higher than the size
of original data. Of course, i've removed all overlap files before
running command below.
# original data
[qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir/g/
1371201660 /data/DC_2013/forcedPhot_csv_dir/g/
# partitioned data
[qserv@clrlsstwn04 ~]$ du -sk
/data/DC_2013/forcedPhot_csv_dir_chunked/g/
1426488452 /data/DC_2013/forcedPhot_csv_dir_chunked/g/
That's why, i've picked two line in both original data set and it
seems their partitioned version have an additional column filled
with a 0 :
[qserv@clrlsstwn04 ~]$ cat l1.txt
117786822259507423,7.3957966992452935,0.89284950462000767,\N,0,745.72313850499529,1429.8763633528101,4.4995131492614746,\N,0.29137018322944641,0,747.31049022025036,1430.0907333904033,\N,\N,\N,0,746.66098764684807,1429.4040414071521,\N,\N,\N,0,0,0,0,0,0,0,0,0,6.1362231689564499,1.385487914154971,2.6747217100805201,20.963277816772461,\N,\N,4.3581457138061523,\N,1.0687153339385986,0,746.66933583824334,1429.9090801332895,\N,\N,\N,0,0,0,0,0,77.895775631472674,29.061176399827357,0,0.898161471,0,-17.386519610881805,57.353877116602995,0,78.180108360892504,25.794014563451498,0,0.999969125,0,1.4593371282074941,57.562767704432233,0,1.0012325,0,746.39238959192767,1429.9768652734542,0,16.061624589279205,2.7097667125873404,3298546023401107,0.045690507375398946,0.00024062812416251493,0.7047580622977917,1755160425,1,53.9074554,51819.376996764448,\N,7.3957966992452935,0.89284950462000767
[qserv@clrlsstwn04 ~]$ cat l1-chunked.txt
117786822259507423 7.3957966992452935 0.89284950462000767
\N 0 745.72313850499529 1429.8763633528101
4.4995131492614746 \N 0.29137018322944641 0
747.31049022025036 1430.0907333904033 \N \N \N 0
746.66098764684807 1429.4040414071521 \N \N \N 0
0 0 0 0 0 0 0 0 6.1362231689564499
1.385487914154971 2.6747217100805201 20.963277816772461
\N \N 4.3581457138061523 \N 1.0687153339385986 0
746.66933583824334 1429.9090801332895\N \N \N 0 0
0 0 0 77.895775631472674 29.061176399827357 0
0.898161471 0 -17.386519610881805 57.353877116602995
0 78.180108360892504 25.794014563451498 0 0.999969125
0 1.4593371282074941 57.562767704432233 0 1.0012325
0 746.39238959192767 1429.9768652734542 0
16.061624589279205 2.7097667125873404 3298546023401107
0.045690507375398946 0.00024062812416251493
0.7047580622977917 1755160425 1 53.9074554
51819.376996764448 \N 7.3957966992452935
0.89284950462000767 0
[qserv@clrlsstwn04 ~]$ cat l2.txt
/data/DC_2013/forcedPhot_csv_dir/g/1755/DeepForcedSource.csv:117786824406993385,12.141404540983455,0.97805062568151802,\N,0,1520.1551503134363,1017.6010715840271,\N,\N,\N,1,1519.8934010903665,1017.8061128666669,\N,\N,\N,0,1520.2042760978757,1016.6587638459159,\N,\N,\N,0,0,0,0,0,0,0,0,0,\N,\N,\N,\N,\N,\N,\N,\N,\N,1,\N,\N,\N,\N,\N,1,1,0,0,0,\N,\N,0,0.921306014,0,-34.836004108190536,57.336590012755131,0,0.54870269981397246,28.522711745077125,0,0.999921203,0,-44.925251112398257,57.449493168762395,0,1.00105381,0,1520.1551503134363,1017.6010715840271,1,20.228827787466521,4.0769451139199271,3113828069935069,\N,\N,\N,1755160457,1,53.9074554,51819.390263564448,\N,12.141404540983455,0.97805062568151802
[qserv@clrlsstwn04 ~]$ cat l2-chunked.txt
117786824406993385 12.141404540983455 0.97805062568151802
\N 0 1520.1551503134363 1017.6010715840271 \N \N
\N 1 1519.8934010903665 1017.8061128666669\N \N \N
0 1520.2042760978757 1016.6587638459159 \N \N \N
0 0 0 0 0 0 0 0 0 \N \N \N \N
\N\N \N \N \N 1 \N \N \N \N \N 1
1 0 0 0 \N \N 0 0.921306014 0
-34.836004108190536 57.33659001275513100.54870269981397246
28.522711745077125 0 0.999921203 0
-44.925251112398257 57.449493168762395 0 1.00105381 0
1520.1551503134363 1017.6010715840271 1
20.228827787466521 4.0769451139199271 3113828069935069
\N \N \N 1755160457 1 53.9074554
51819.390263564448 \N 12.141404540983455
0.978050625681518020 0
Do you think there could have a problem in my partitioner
configuration ?
(cf. https://github.com/fjammes/misc/tree/master/qserv_partitioner/DC_2013)
Thanks for your help and have a nice day,
Fabrice
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
