 |
 |
Hello Serge, It seems the partitioning process runned successfully, and Osman Aidel, our CC-IN2P3 database expert, is ready to start loading it with MariaDB. Nevertheless the produced data size is slighty higher than the size of original data. Of course, i've removed all overlap files before running command below. # original data [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir/g/ *1371201660* /data/DC_2013/forcedPhot_csv_dir/g/ # partitioned data [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir_chunked/g/ *1426488452* /data/DC_2013/forcedPhot_csv_dir_chunked/g/ That's why, i've picked two line in both original data set and it seems their partitioned version have an additional column filled with a 0 : [qserv@clrlsstwn04 ~]$ cat l1.txt 117786822259507423,7.3957966992452935,0.89284950462000767,\N,0,745.72313850499529,1429.8763633528101,4.4995131492614746,\N,0.29137018322944641,0,747.31049022025036,1430.0907333904033,\N,\N,\N,0,746.66098764684807,1429.4040414071521,\N,\N,\N,0,0,0,0,0,0,0,0,0,6.1362231689564499,1.385487914154971,2.6747217100805201,20.963277816772461,\N,\N,4.3581457138061523,\N,1.0687153339385986,0,746.66933583824334,1429.9090801332895,\N,\N,\N,0,0,0,0,0,77.895775631472674,29.061176399827357,0,0.898161471,0,-17.386519610881805,57.353877116602995,0,78.180108360892504,25.794014563451498,0,0.999969125,0,1.4593371282074941,57.562767704432233,0,1.0012325,0,746.39238959192767,1429.9768652734542,0,16.061624589279205,2.7097667125873404,3298546023401107,0.045690507375398946,0.00024062812416251493,0.7047580622977917,1755160425,1,53.9074554,51819.376996764448,\N,7.3957966992452935,0.89284950462000767 [qserv@clrlsstwn04 ~]$ cat l1-chunked.txt 117786822259507423 7.3957966992452935 0.89284950462000767 \N 0 745.72313850499529 1429.8763633528101 4.4995131492614746 \N 0.29137018322944641 0 747.31049022025036 1430.0907333904033 \N \N \N 0 746.66098764684807 1429.4040414071521 \N \N \N 0 0 0 0 0 0 0 0 0 6.1362231689564499 1.385487914154971 2.6747217100805201 20.963277816772461 \N \N 4.3581457138061523 \N 1.0687153339385986 0 746.66933583824334 1429.9090801332895\N \N \N 0 0 0 0 0 77.895775631472674 29.061176399827357 0 0.898161471 0 -17.386519610881805 57.353877116602995 0 78.180108360892504 25.794014563451498 0 0.999969125 0 1.4593371282074941 57.562767704432233 0 1.0012325 0 746.39238959192767 1429.9768652734542 0 16.061624589279205 2.7097667125873404 3298546023401107 0.045690507375398946 0.00024062812416251493 0.7047580622977917 1755160425 1 53.9074554 51819.376996764448 \N 7.3957966992452935 0.89284950462000767 *0* [qserv@clrlsstwn04 ~]$ cat l2.txt /data/DC_2013/forcedPhot_csv_dir/g/1755/DeepForcedSource.csv:117786824406993385,12.141404540983455,0.97805062568151802,\N,0,1520.1551503134363,1017.6010715840271,\N,\N,\N,1,1519.8934010903665,1017.8061128666669,\N,\N,\N,0,1520.2042760978757,1016.6587638459159,\N,\N,\N,0,0,0,0,0,0,0,0,0,\N,\N,\N,\N,\N,\N,\N,\N,\N,1,\N,\N,\N,\N,\N,1,1,0,0,0,\N,\N,0,0.921306014,0,-34.836004108190536,57.336590012755131,0,0.54870269981397246,28.522711745077125,0,0.999921203,0,-44.925251112398257,57.449493168762395,0,1.00105381,0,1520.1551503134363,1017.6010715840271,1,20.228827787466521,4.0769451139199271,3113828069935069,\N,\N,\N,1755160457,1,53.9074554,51819.390263564448,\N,12.141404540983455,0.97805062568151802 [qserv@clrlsstwn04 ~]$ cat l2-chunked.txt 117786824406993385 12.141404540983455 0.97805062568151802 \N 0 1520.1551503134363 1017.6010715840271 \N \N \N 1 1519.8934010903665 1017.8061128666669\N \N \N 0 1520.2042760978757 1016.6587638459159 \N \N \N 0 0 0 0 0 0 0 0 0 \N \N \N \N \N\N \N \N \N 1 \N \N \N \N \N 1 1 0 0 0 \N \N 0 0.921306014 0 -34.836004108190536 57.33659001275513100.54870269981397246 28.522711745077125 0 0.999921203 0 -44.925251112398257 57.449493168762395 0 1.00105381 0 1520.1551503134363 1017.6010715840271 1 20.228827787466521 4.0769451139199271 3113828069935069 \N \N \N 1755160457 1 53.9074554 51819.390263564448 \N 12.141404540983455 0.978050625681518020 *0* Do you think there could have a problem in my partitioner configuration ? (cf. https://github.com/fjammes/misc/tree/master/qserv_partitioner/DC_2013) Thanks for your help and have a nice day, Fabrice ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1