 |
 |
Hi All, I submitted about half of pass2 jobs: about 23% of jobs failed because of "Error writing LCIO file" exception caused by "No space left on device" The requested disk size is more that total files sizes. I will make a CCPR soon, it seems to be a farm related problem 2018-12-15 07:28:36 [INFO] org.lcsim.job.EventPrintLoopAdapter recordSupplied :: event: 55082949; time: 1461394849551037316; seq: 375000 Exception in thread "main" java.lang.RuntimeException: Error writing LCIO file at org.lcsim.util.loop.LCIODriver.process(LCIODriver.java:116) at org.lcsim.util.Driver.doProcess(Driver.java:261) at org.lcsim.util.Driver.processChildren(Driver.java:271) at org.lcsim.util.Driver.process(Driver.java:187) at org.lcsim.util.DriverAdapter.recordSupplied(DriverAdapter.java:74) at org.lcsim.job.JobControlManager.processEvent(JobControlManager.java:819) at org.hps.evio.EvioToLcio.run(EvioToLcio.java:618) at org.hps.evio.EvioToLcio.main(EvioToLcio.java:92) Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:315) at hep.io.xdr.XDROutputStream$CountedOutputStream.write(XDROutputStream.java:103) at java.io.DataOutputStream.write(DataOutputStream.javaError in <TBranchElement::Fill>: Failed filling branch:tracks.fs_particle, nbytes=-1 - Some fles have this exception but this is not fatal exception, i.e. reconstruction is not stopped 2018-12-15 07:20:26 [INFO] org.hps.evio.AugmentedSvtEvioReader processSvtHeaders :: Caught 5 SvtEvioHeaderExceptions for event 17654220 of 4 types: SvtEvioHeaderMultisampleErrorBitException SvtEvioHeaderApvBufferAddressException SvtEvioHeaderApvFrameCountException SvtEvioHeaderApvReadErrorException Files from the run 7988 show the following exception, NOTE this exception happens only for files from the run 7988 hps_007988.0_v11_18_18_Recon.err:java.lang.NumberFormatException: For input string: "18GTP_CLUSTER_PULSE_COIN" hps_007988.0_v11_18_18_Recon.err: at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) If you would like to look into some of outputs, the work directory is the following: /work/hallb/hps/data/physrun2016/pass2 Some of files are already in the tar and some not yet, I will not submit rest of jobs, will wait for the CCPR response. Rafo ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the HPS-SOFTWARE list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1