 |
 |
Hello Serge, Thanks for your answer, it helps a lot. Have a nice day, Fabrice On 14/02/2014 19:43, Serge Monkewitz wrote: > Fabrice - > > That’s expected. This extra field is the subChunkId computed by the > partitioner. Given your configuration, it should always be zero (since > you’ve sized your sub-chunks to be the size of entire chunks). You can > ignore that field while loading. > > Cheers, > Serge > > On Feb 14, 2014, at 8:49 AM, Fabrice Jammes <[log in to unmask] > <mailto:[log in to unmask]>> wrote: > >> Hello Serge, >> >> It seems the partitioning process runned successfully, and Osman >> Aidel, our CC-IN2P3 database expert, is ready to start loading it >> with MariaDB. >> Nevertheless the produced data size is slighty higher than the size >> of original data. Of course, i've removed all overlap files before >> running command below. >> >> # original data >> [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir/g/ >> *1371201660* /data/DC_2013/forcedPhot_csv_dir/g/ >> >> # partitioned data >> [qserv@clrlsstwn04 ~]$ du -sk /data/DC_2013/forcedPhot_csv_dir_chunked/g/ >> *1426488452* /data/DC_2013/forcedPhot_csv_dir_chunked/g/ >> >> That's why, i've picked two line in both original data set and it >> seems their partitioned version have an additional column filled with >> a 0 : >> >> [qserv@clrlsstwn04 ~]$ cat l1.txt >> 117786822259507423,7.3957966992452935,0.89284950462000767,\N,0,745.72313850499529,1429.8763633528101,4.4995131492614746,\N,0.29137018322944641,0,747.31049022025036,1430.0907333904033,\N,\N,\N,0,746.66098764684807,1429.4040414071521,\N,\N,\N,0,0,0,0,0,0,0,0,0,6.1362231689564499,1.385487914154971,2.6747217100805201,20.963277816772461,\N,\N,4.3581457138061523,\N,1.0687153339385986,0,746.66933583824334,1429.9090801332895,\N,\N,\N,0,0,0,0,0,77.895775631472674,29.061176399827357,0,0.898161471,0,-17.386519610881805,57.353877116602995,0,78.180108360892504,25.794014563451498,0,0.999969125,0,1.4593371282074941,57.562767704432233,0,1.0012325,0,746.39238959192767,1429.9768652734542,0,16.061624589279205,2.7097667125873404,3298546023401107,0.045690507375398946,0.00024062812416251493,0.7047580622977917,1755160425,1,53.9074554,51819.376996764448,\N,7.3957966992452935,0.89284950462000767 >> >> [qserv@clrlsstwn04 ~]$ cat l1-chunked.txt >> 117786822259507423 7.3957966992452935 0.89284950462000767 \N >> 0 745.72313850499529 1429.8763633528101 4.4995131492614746 >> \N 0.29137018322944641 0 747.31049022025036 >> 1430.0907333904033 \N \N \N 0 746.66098764684807 >> 1429.4040414071521 \N \N \N 0 0 0 0 0 0 >> 0 0 0 6.1362231689564499 1.385487914154971 >> 2.6747217100805201 20.963277816772461 \N \N >> 4.3581457138061523 \N 1.0687153339385986 0 >> 746.66933583824334 1429.9090801332895\N \N \N 0 0 0 >> 0 0 77.895775631472674 29.061176399827357 0 >> 0.898161471 0 -17.386519610881805 57.353877116602995 0 >> 78.180108360892504 25.794014563451498 0 0.999969125 0 >> 1.4593371282074941 57.562767704432233 0 1.0012325 0 >> 746.39238959192767 1429.9768652734542 0 16.061624589279205 >> 2.7097667125873404 3298546023401107 0.045690507375398946 >> 0.00024062812416251493 0.7047580622977917 1755160425 1 >> 53.9074554 51819.376996764448 \N 7.3957966992452935 >> 0.89284950462000767 *0* >> >> [qserv@clrlsstwn04 ~]$ cat l2.txt >> /data/DC_2013/forcedPhot_csv_dir/g/1755/DeepForcedSource.csv:117786824406993385,12.141404540983455,0.97805062568151802,\N,0,1520.1551503134363,1017.6010715840271,\N,\N,\N,1,1519.8934010903665,1017.8061128666669,\N,\N,\N,0,1520.2042760978757,1016.6587638459159,\N,\N,\N,0,0,0,0,0,0,0,0,0,\N,\N,\N,\N,\N,\N,\N,\N,\N,1,\N,\N,\N,\N,\N,1,1,0,0,0,\N,\N,0,0.921306014,0,-34.836004108190536,57.336590012755131,0,0.54870269981397246,28.522711745077125,0,0.999921203,0,-44.925251112398257,57.449493168762395,0,1.00105381,0,1520.1551503134363,1017.6010715840271,1,20.228827787466521,4.0769451139199271,3113828069935069,\N,\N,\N,1755160457,1,53.9074554,51819.390263564448,\N,12.141404540983455,0.97805062568151802 >> [qserv@clrlsstwn04 ~]$ cat l2-chunked.txt >> 117786824406993385 12.141404540983455 0.97805062568151802 \N >> 0 1520.1551503134363 1017.6010715840271 \N \N \N 1 >> 1519.8934010903665 1017.8061128666669\N \N \N 0 >> 1520.2042760978757 1016.6587638459159 \N \N \N 0 0 >> 0 0 0 0 0 0 0 \N \N \N \N \N\N \N >> \N \N 1 \N \N \N \N \N 1 1 0 0 0 >> \N \N 0 0.921306014 0 -34.836004108190536 >> 57.33659001275513100.54870269981397246 28.522711745077125 0 >> 0.999921203 0 -44.925251112398257 57.449493168762395 0 >> 1.00105381 0 1520.1551503134363 1017.6010715840271 1 >> 20.228827787466521 4.0769451139199271 3113828069935069 \N \N >> \N 1755160457 1 53.9074554 51819.390263564448 \N >> 12.141404540983455 0.978050625681518020 *0* >> >> Do you think there could have a problem in my partitioner configuration ? >> (cf. >> https://github.com/fjammes/misc/tree/master/qserv_partitioner/DC_2013) >> >> Thanks for your help and have a nice day, >> >> Fabrice >> >> ------------------------------------------------------------------------ >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1