 |
 |
Fabrice - That’s expected. This extra field is the subChunkId computed by the partitioner. Given your configuration, it should always be zero (since you’ve sized your sub-chunks to be the size of entire chunks). You can ignore that field while loading. Cheers, Serge On Feb 14, 2014, at 8:49 AM, Fabrice Jammes <[log in to unmask]> wrote: > Hello Serge, > > It seems the partitioning process runned successfully, and Osman Aidel, our CC-IN2P3 database expert, is ready to start loading it with MariaDB. > Nevertheless the produced data size is slighty higher than the size of original data. Of course, i've removed all overlap files before running command below. > > # original data > [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir/g/ > 1371201660 /data/DC_2013/forcedPhot_csv_dir/g/ > > # partitioned data > [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir_chunked/g/ > 1426488452 /data/DC_2013/forcedPhot_csv_dir_chunked/g/ > > That's why, i've picked two line in both original data set and it seems their partitioned version have an additional column filled with a 0 : > > [qserv@clrlsstwn04 ~]$ cat l1.txt > 117786822259507423,7.3957966992452935,0.89284950462000767,\N,0,745.72313850499529,1429.8763633528101,4.4995131492614746,\N,0.29137018322944641,0,747.31049022025036,1430.0907333904033,\N,\N,\N,0,746.66098764684807,1429.4040414071521,\N,\N,\N,0,0,0,0,0,0,0,0,0,6.1362231689564499,1.385487914154971,2.6747217100805201,20.963277816772461,\N,\N,4.3581457138061523,\N,1.0687153339385986,0,746.66933583824334,1429.9090801332895,\N,\N,\N,0,0,0,0,0,77.895775631472674,29.061176399827357,0,0.898161471,0,-17.386519610881805,57.353877116602995,0,78.180108360892504,25.794014563451498,0,0.999969125,0,1.4593371282074941,57.562767704432233,0,1.0012325,0,746.39238959192767,1429.9768652734542,0,16.061624589279205,2.7097667125873404,3298546023401107,0.045690507375398946,0.00024062812416251493,0.7047580622977917,1755160425,1,53.9074554,51819.376996764448,\N,7.3957966992452935,0.89284950462000767 > > [qserv@clrlsstwn04 ~]$ cat l1-chunked.txt > 117786822259507423 7.3957966992452935 0.89284950462000767 \N 0 745.72313850499529 1429.8763633528101 4.4995131492614746 \N 0.29137018322944641 0 747.31049022025036 1430.0907333904033 \N \N \N 0 746.66098764684807 1429.4040414071521 \N \N \N 0 0 0 0 0 0 0 0 0 6.1362231689564499 1.385487914154971 2.6747217100805201 20.963277816772461 \N \N 4.3581457138061523 \N 1.0687153339385986 0 746.66933583824334 1429.9090801332895\N \N \N 0 0 0 0 0 77.895775631472674 29.061176399827357 0 0.898161471 0 -17.386519610881805 57.353877116602995 0 78.180108360892504 25.794014563451498 0 0.999969125 0 1.4593371282074941 57.562767704432233 0 1.0012325 0 746.39238959192767 1429.9768652734542 0 16.061624589279205 2.7097667125873404 3298546023401107 0.045690507375398946 0.00024062812416251493 0.7047580622977917 1755160425 1 53.9074554 51819.376996764448 \N 7.3957966992452935 0.89284950462000767 0 > > [qserv@clrlsstwn04 ~]$ cat l2.txt > /data/DC_2013/forcedPhot_csv_dir/g/1755/DeepForcedSource.csv:117786824406993385,12.141404540983455,0.97805062568151802,\N,0,1520.1551503134363,1017.6010715840271,\N,\N,\N,1,1519.8934010903665,1017.8061128666669,\N,\N,\N,0,1520.2042760978757,1016.6587638459159,\N,\N,\N,0,0,0,0,0,0,0,0,0,\N,\N,\N,\N,\N,\N,\N,\N,\N,1,\N,\N,\N,\N,\N,1,1,0,0,0,\N,\N,0,0.921306014,0,-34.836004108190536,57.336590012755131,0,0.54870269981397246,28.522711745077125,0,0.999921203,0,-44.925251112398257,57.449493168762395,0,1.00105381,0,1520.1551503134363,1017.6010715840271,1,20.228827787466521,4.0769451139199271,3113828069935069,\N,\N,\N,1755160457,1,53.9074554,51819.390263564448,\N,12.141404540983455,0.97805062568151802 > [qserv@clrlsstwn04 ~]$ cat l2-chunked.txt > 117786824406993385 12.141404540983455 0.97805062568151802 \N 0 1520.1551503134363 1017.6010715840271 \N \N \N 1 1519.8934010903665 1017.8061128666669\N \N \N 0 1520.2042760978757 1016.6587638459159 \N \N \N 0 0 0 0 0 0 0 0 0 \N \N \N \N \N\N \N \N \N 1 \N \N \N \N \N 1 1 0 0 0 \N \N 0 0.921306014 0 -34.836004108190536 57.33659001275513100.54870269981397246 28.522711745077125 0 0.999921203 0 -44.925251112398257 57.449493168762395 0 1.00105381 0 1520.1551503134363 1017.6010715840271 1 20.228827787466521 4.0769451139199271 3113828069935069 \N \N \N 1755160457 1 53.9074554 51819.390263564448 \N 12.141404540983455 0.978050625681518020 0 > > Do you think there could have a problem in my partitioner configuration ? > (cf. https://github.com/fjammes/misc/tree/master/qserv_partitioner/DC_2013) > > Thanks for your help and have a nice day, > > Fabrice > > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1