 |
 |
Hello Serge, Thanks again for your help. I looked better at the log files and discover that the partitioner has failed because it doesn't have read access on a subset of the DC2013 files. I should have had a deeper look at these log files before asking my question, sorry. Then, I set the correct access rights, improved a bit the error recovery feature in my script, and then relaunch it. So, currently, it has produced 1TB of data, and is still running, but without error. I will let you know soon if the partitionning was successfull. For the future, I can create you an account on clrlsstwn04.in2p3.fr if you want, but i need the IP adress of the machine were you'll run the ssh client in order our sysadmin set the correct ACL on our firewall. Have a nice day, Fabrice On 02/04/2014 05:06 AM, Serge Monkewitz wrote: > Fabrice, > > There are only 2 ways I can see how the size could decrease. The first > is if the input represents NULL values as “NULL”, and you configure > the partitioner to output “\N” instead. The second is if you configure > the partitioner to drop columns. > > By the way - I cannot look at the files you mention below. If I ssh to > ccage.in2p3.fr <http://ccage.in2p3.fr>, then ssh qserv@clrlsstwn04, I > get the following error message: > ssh: connect to host clrlsstwn04 port 22: No route to host > > If you could tell me where your partitioner config files are, which > machine I should login to, and the full partitioner command line > invocation, I will take a look. > > Cheers, > Serge > > On Jan 31, 2014, at 2:04 AM, Fabrice Jammes <[log in to unmask] > <mailto:[log in to unmask]>> wrote: > >> Hello Serge, >> >> After having downloaded again DC2013 RunDeepForcedSource csv files, >> partitionning finally runnned with success. >> >> Nevertheless the size of the produced data, overlap files included, >> is smaller than the size of the original data. >> >> Do you think this can be possible, or i shall investigate for a >> possible new problem ? >> >> # Original data: >> [qserv@clrlsstwn04 DC_2013]$ du -skh /data/DC_2013/forcedPhot_csv_dir/g/ >> 1.4T/data/DC_2013/forcedPhot_csv_dir/g/ >> >> # Partitionned data >> [qserv@clrlsstwn04 DC_2013]$ du -skh >> /data/DC_2013/forcedPhot_csv_dir_chunked/ >> 908G/data/DC_2013/forcedPhot_csv_dir_chunked/ >> >> Thanks for your answer, and have a nice day. >> >> Fabrice >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1