Print

Print
Hello Serge,

Thanks again for your help.

I looked better at the log files and discover that the partitioner has failed because it doesn't have read access on a subset of the DC2013 files. I should have had a deeper look at these log files before asking my question, sorry.
Then, I set the correct access rights, improved a bit the error recovery feature in my script, and then relaunch it. So, currently, it has produced 1TB of data, and is still running, but without error. I will let you know soon
if the partitionning was successfull.

For the future, I can create you an account on clrlsstwn04.in2p3.fr if you want, but i need the IP adress of the machine were you'll run the ssh client in order our sysadmin set the correct ACL on our firewall.

Have a nice day,

Fabrice

On 02/04/2014 05:06 AM, Serge Monkewitz wrote:
[log in to unmask]" type="cite">
Fabrice,

There are only 2 ways I can see how the size could decrease. The first is if the input represents NULL values as “NULL”, and you configure the partitioner to output “\N” instead. The second is if you configure the partitioner to drop columns.

By the way - I cannot look at the files you mention below. If I ssh to ccage.in2p3.fr, then ssh qserv@clrlsstwn04, I get the following error message:
ssh: connect to host clrlsstwn04 port 22: No route to host

If you could tell me where your partitioner config files are, which machine I should login to, and the full partitioner command line invocation, I will take a look.

Cheers,
Serge

On Jan 31, 2014, at 2:04 AM, Fabrice Jammes <[log in to unmask]> wrote:

Hello Serge,

After having downloaded again DC2013 RunDeepForcedSource csv files, partitionning finally runnned with success.

Nevertheless the size of the produced data, overlap files included, is smaller than the size of the original data.

Do you think this can be possible, or i shall investigate for a possible new problem ?

# Original data:
[qserv@clrlsstwn04 DC_2013]$ du -skh /data/DC_2013/forcedPhot_csv_dir/g/
1.4T /data/DC_2013/forcedPhot_csv_dir/g/

# Partitionned data
[qserv@clrlsstwn04 DC_2013]$ du -skh /data/DC_2013/forcedPhot_csv_dir_chunked/
908G /data/DC_2013/forcedPhot_csv_dir_chunked/

Thanks for your answer, and have a nice day.

Fabrice

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1



Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1