Hi Stepphen,
It's clear that Nurcan has done quite a bit of work to digest some of this information. Do you know if (subset of) this information is graphed somewhere? I am curious about long-term trends of failure rate, distribution of failure causes, etc. Cheers.
Charlie
--
Charles C. Young
M.S. 43, Stanford Linear Accelerator Center
P.O. Box 20450
Stanford, CA 94309
[log in to unmask]
voice (650) 926 2669
fax (650) 926 2923
CERN GSM +41 76 487 2069
> -----Original Message-----
> From: [log in to unmask]
> [mailto:[log in to unmask]] On
> Behalf Of Stephen J. Gowdy
> Sent: Thursday, August 17, 2006 12:19 AM
> To: atlas-sccs-planning-l
> Subject: [Usatlas-prodsys-l] Panda shift report August 14-15,
> 2006 (fwd)
>
> FYI (to do with discussion of success rate for jobs).
>
> --
> /------------------------------------+-------------------------\
> |Stephen J. Gowdy, SLAC | CERN Office: 32-2-A22|
> |http://www.slac.stanford.edu/~gowdy/ | CH-1211 Geneva 23 |
> |http://calendar.yahoo.com/gowdy | Switzerland |
> |EMail: [log in to unmask] | Tel: +41 22 767 5840 |
> \------------------------------------+-------------------------/
>
> ---------- Forwarded message ----------
> Date: Wed, 16 Aug 2006 16:00:11 -0500 (CDT)
> From: Nurcan Ozturk <[log in to unmask]>
> To: [log in to unmask]
> Subject: [Usatlas-prodsys-l] Panda shift report August 14-15, 2006
>
> Hi all,
>
> Here is the Panda production status of the last 2 days:
>
> Wed Aug 16 10:58:15 2006 Central
> --------------------------------------------------------------
> ----------------------------------------------
> All CEs and jobs. Show production, analysis, test, all jobs/CEs
> --------------------------------------------------------------
> ----------------------------------------------
> Job wall time: 30948 hrs Error losses: trans: 2219 (7.2%)
> panda: 832 (2.7%) ddm: 68 (0.2%) other: 506 (1.6%)
> --------------------------------------------------------------
> -----------------------------------------------
> Error type (type count) Count CPU-hrs Latest Code:
> Description
> --------------------------------------------------------------
> -----------------------------------------------
> All defined:1 assigned:161 waiting:474 activated:1010
> running:752 finished:2339 failed:724 (23.6%)
>
> ddmErrorCode (9) 8 0.0 08-15 01:24 100:
> Input file GUID not found or input prodDBlock not accessible
> ddmErrorCode (9) 1 11.9 08-16 07:14 200:
> Could not add output files to dataset
> jobDispatcherErrorCode (59) 59 698.9 08-16
> 01:01 100: Lost heartbeat
> pilotErrorCode (251) 174 0.9 08-16 04:28
> 1099: DQ2 staging input file failed
> pilotErrorCode (251) 1 0.1 08-14 17:06
> 1132: Saving output files to DDM area returned non-zero code
> pilotErrorCode (251) 6 55.2 08-16 04:26
> 1142: DQ2 put error: failed to register the file on local SE
> pilotErrorCode (251) 68 505.9 08-16 11:21
> 1150: Looping job killed by pilot
> pilotErrorCode (251) 2 133.3 08-15 13:58
> 1200: Job killed by SIGTERM from batch system or Condor (eg
> walltime limit)
> taskBufferErrorCode (149) 149 0.0 08-15 19:22
> 100: Job expired and killed six days after submission (or
> killed by user)
> transExitCode (234) 72 13.0 08-16 04:30
> 1: Unspecified error, consult log file
> transExitCode (234) 11 0.2 08-16 03:51
> 134: Athena core dump or timeout, or conddb DB connect exception
> transExitCode (234) 1 76.8 08-14 12:00
> 143: Unknown error code
> transExitCode (234) 8 6.4 08-16 10:30
> 2: Athena core dump
> transExitCode (234) 35 26.5 08-16 06:23
> 40: Athena crash - consult log file
> transExitCode (234) 14 89.8 08-14 12:08
> 41: TRF_OUTFILE - output file not found
> transExitCode (234) 2 75.7 08-14 19:18
> 50: Athena crash - consult log file
> transExitCode (234) 17 818.9 08-16 11:31
> 60: TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)
> transExitCode (234) 74 1111.3 08-15 18:12
> 99: TRF_UNKNOWN - unknown transformation error
> --------------------------------------------------------------
> --------------------------------------------
> ANALY_BNL_ATLAS_1 defined:0 assigned:0 waiting:0
> activated:0 running:0 finished:98 failed:18 (15.5%)
>
> jobDispatcherErrorCode (1) 1 1.1 08-14 15:22
> 100: Lost heartbeat
> pilotErrorCode (1) 1 0.1 08-14 17:06 1132:
> Saving output files to DDM area returned non-zero code
> transExitCode (16) 16 2.5 08-16 06:23 40:
> Athena crash - consult log file
> --------------------------------------------------------------
> --------------------------------------------
> ANALY_BNL_ATLAS_2 defined:1 assigned:0 waiting:0
> activated:1 running:0 finished:0 failed:0
> --------------------------------------------------------------
> --------------------------------------------
> ANALY_LONG_BNL_ATLAS defined:0 assigned:0 waiting:0
> activated:0 running:0 finished:0 failed:19
>
> transExitCode (19) 19 24.0 08-16 04:36 40:
> Athena crash - consult log file
> --------------------------------------------------------------
> -------------------------------------------
> ANALY_UTA-DPCC
> --------------------------------------------------------------
> -------------------------------------------
> BNL_ATLAS_1 defined:0 assigned:0 waiting:0
> activated:343 running:283 finished:843 failed:162 (16.1%)
>
> jobDispatcherErrorCode (48) 48 53.9 08-16
> 01:01 100: Lost heartbeat
> pilotErrorCode (68) 68 505.9 08-16 11:21
> 1150: Looping job killed by pilot
> taskBufferErrorCode (19) 19 0.0 08-15 17:44
> 100: Job expired and killed six days after submission (or
> killed by user)
> transExitCode (27) 27 389.8 08-15 18:12 99:
> TRF_UNKNOWN - unknown transformation error
> --------------------------------------------------------------
> -------------------------------------------
> BNL_ATLAS_2
> --------------------------------------------------------------
> ------------------------------------------
> BU_ATLAS_Tier2 defined:0 assigned:0 waiting:0
> activated:92 running:93 finished:124 failed:41 (24.8%)
>
> pilotErrorCode (1) 1 66.7 08-15 02:44 1200:
> Job killed by SIGTERM from batch system or Condor (eg walltime limit)
> transExitCode (40) 40 8.1 08-15 20:43 1:
> Unspecified error, consult log file
> --------------------------------------------------------------
> ------------------------------------------
> BU_ATLAS_Tier2o defined:0 assigned:7 waiting:0
> activated:10 running:12 finished:20 failed:2 (9.1%)
> transExitCode (2) 2 0.3 08-14 11:59 1:
> Unspecified error, consult log file
> --------------------------------------------------------------
> ------------------------------------------
> IU_ATLAS_Tier2 defined:0 assigned:10 waiting:0
> activated:67 running:64 finished:157 failed:28 (15.1%)
> transExitCode (28) 7 0.7 08-16 06:21 2:
> Athena core dump
> transExitCode (28) 6 289.2 08-16 11:05 60:
> TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)
> transExitCode (28) 15 156.6 08-14 22:33 99:
> TRF_UNKNOWN - unknown transformation error
> --------------------------------------------------------------
> -------------------------------------------
> Unassigned defined:0 assigned:0 waiting:474
> activated:0 running:0 finished:0 failed:129
>
> taskBufferErrorCode (129) 129 0.0 08-15 19:22
> 100: Job expired and killed six days after submission (or
> killed by user)
> --------------------------------------------------------------
> -------------------------------------------
> OU_OCHEP_SWT2 defined:0 assigned:6 waiting:0
> activated:116 running:81 finished:279 failed:40 (12.5%)
>
> jobDispatcherErrorCode (5) 5 359.0 08-16 00:40
> 100: Lost heartbeat
> taskBufferErrorCode (1) 1 0.0 08-15 17:41
> 100: Job expired and killed six days after submission (or
> killed by user)
> transExitCode (34) 13 1.7 08-16 04:30 1:
> Unspecified error, consult log file
> transExitCode (34) 1 76.8 08-14 12:00 143:
> Unknown error code
> transExitCode (34) 1 89.5 08-14 11:54 41:
> TRF_OUTFILE - output file not found
> transExitCode (34) 1 64.1 08-14 12:01 50:
> Athena crash - consult log file
> transExitCode (34) 2 96.3 08-16 10:29 60:
> TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)
> transExitCode (34) 16 234.9 08-15 13:07 99:
> TRF_UNKNOWN - unknown transformation error
> --------------------------------------------------------------
> --------------------------------------------
> PROD_SLAC defined:0 assigned:41 waiting:0
> activated:0 running:0 finished:6 failed:0 (0.0%)
> --------------------------------------------------------------
> --------------------------------------------
> UC_ATLAS_MWT2 defined:0 assigned:22 waiting:0
> activated:101 running:56 finished:149 failed:4 (2.6%)
>
> ddmErrorCode (1) 1 11.9 08-16 07:14 200:
> Could not add output files to dataset
> transExitCode (3) 1 11.6 08-14 19:18 50:
> Athena crash - consult log file
> transExitCode (3) 2 96.3 08-15 20:38 60:
> TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)
> --------------------------------------------------------------
> --------------------------------------------
> UC_Teraport defined:0 assigned:12 waiting:0
> activated:183 running:64 finished:321 failed:202 (38.6%)
>
> jobDispatcherErrorCode (1) 1 66.7 08-15 08:08
> 100: Lost heartbeat
> pilotErrorCode (180) 174 0.9 08-16 04:28
> 1099: DQ2 staging input file failed
> pilotErrorCode (180) 6 55.2 08-16 04:26
> 1142: DQ2 put error: failed to register the file on local SE
> transExitCode (21) 7 1.4 08-15 03:15 1:
> Unspecified error, consult log file
> transExitCode (21) 1 5.6 08-16 10:30 2:
> Athena core dump
> transExitCode (21) 13 0.3 08-14 12:08 41:
> TRF_OUTFILE - output file not found
> --------------------------------------------------------------
> ---------------------------------------------
> UTA-DPCC defined:0 assigned:61 waiting:0
> activated:97 running:99 finished:342 failed:57 (14.3%)
>
> ddmErrorCode (8) 8 0.0 08-15 01:24 100:
> Input file GUID not found or input prodDBlock not accessible
> jobDispatcherErrorCode (4) 4 218.2 08-16 01:01
> 100: Lost heartbeat
> pilotErrorCode (1) 1 66.7 08-15 13:58 1200:
> Job killed by SIGTERM from batch system or Condor (eg walltime limit)
> transExitCode (44) 10 1.4 08-15 06:22 1:
> Unspecified error, consult log file
> transExitCode (44) 11 0.2 08-16 03:51 134:
> Athena core dump or timeout, or conddb DB connect exception
> transExitCode (44) 7 337.2 08-16 11:31 60:
> TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)
> transExitCode (44) 16 330.0 08-15 11:58 99:
> TRF_UNKNOWN - unknown transformation error
> --------------------------------------------------------------
> --------------------------------------------
> UTA_SWT2 defined:0 assigned:2 waiting:0
> activated:0 running:0 finished:0 failed:0
> --------------------------------------------------------------
> --------------------------------------------
>
>
> The pilot job status from the submit host:
> ------------------------------------------------------------------
> [sm@atlas002 jobscheduler]$ ./queue-summary.py
> ==================== Condor Queue Summary
> ==================== condor_q run at Wed Aug 16 10:58:15 2006
> Maximum jobs on a remote host (all but UNKNOWN & UNSUBMITTED): 200
> Maximum jobs being sent to remote host: 5
>
> atlas.bu.edu
> PENDING 59
> ACTIVE 105
>
> atlas.dpcc.uta.edu
> PENDING 100
> ACTIVE 100
> UNSUBMITTED 9
>
> atlas.iu.edu
> PENDING 52
> ACTIVE 60
>
> gk01.swt2.uta.edu
> PENDING 5
> ACTIVE 1
> STAGE_OUT 2
>
> osgserv01.slac.stanford.edu
> PENDING 52
> STAGE_OUT 1
>
> tier2-01.ochep.ou.edu
> PENDING 50
> ACTIVE 80
>
> tier2-osg.uchicago.edu
> PENDING 52
> ACTIVE 41
>
> tp-osg.uchicago.edu
> PENDING 54
> ACTIVE 63
>
> ------------------------------------------------------------------
>
> Some notes:
>
> 1. 51 jobs of the type
> csc11.005538.AlpgenJimmyToplnlnNp3.evgen.v11004211
> failed with "transExitCode=1:Unspecified error, consult log
> file" in their second and third attempt, due to:
>
> -------- Problem report -------
> [Unknown Problem]
> !!! AthenaEventLoo ERROR Terminating event processing loop
> due to errors!!!
> ================================
>
> 43 jobs of the same type failed with "lost heartbeat" due to
> the same reason. I opened a Savannah bug #19047.
>
> 2. 6 jobs of the type csc11.005200.T1_McAtNlo_Jimmy.digit.v11004205
> failed at UTA-DPCC with "transExitCode=1:Unspecified error,
> consult log file" due to:
> --------------------------------------------------------------
> -----------------
> G4AtlasAlg: Event Nr. 1 start processing
> EA2F6442-1926-DB11-9647-00123F20A423 Error Cannot open container,
> invalid Database handle.
> StorageSvc Error The requested
> container:POOLContainer_McEventCollection cannot be opened!
>
> *** Break *** segmentation violation
> Generating stack trace...
> /usr/bin/addr2line: python: No such file or directory
> /usr/bin/addr2line: python: No such file or directory
> 0x03d4373e in
> FadsGeneratorT<AthenaHepMCInterface>::GenerateAnEvent() +
> 0x1e from
> /data73/grid3-1.1.11/apps/atlas_app/atlas_rel/11.0.42/dist/11.
> 0.42/InstallArea/
> i686-slc3-gcc323-opt/
> --------------------------------------------------------------
> --------------------
>
> I opened a Savannah bug #19104.
>
> 3. 11 jobs of the type
> csc11.005250.McAtNloWminenu.evgen.v11004209 failed at
> UTA-DPCC with "transExitCode=134: Athena core dump or
> timeout, or conddb DB connect exception" in the third attempt due to:
> --------------------------------------------------------------
> --------------------
> found 258 particles
> AtRndmGenSvc INFO Initializing AtRndmGenSvc - package version
> AthenaServices-01-07-27
> INITIALISING RANDOM NUMBER STREAMS.
>
>
> HERWIG 6.507 8th March 2005
>
> Please reference: G. Marchesini, B.R. Webber,
> G.Abbiendi, I.G.Knowles, M.H.Seymour & L.Stanco
> Computer Physics Communications 67 (1992) 465
> and
> G.Corcella, I.G.Knowles, G.Marchesini, S.Moretti,
> K.Odagiri, P.Richardson, M.H.Seymour & B.R.Webber,
> JHEP 0101 (2001) 010
> fmt: end of file
> apparent state: unit 61 named mcatnlo31.005250.Wminenu._000020.events
> last format: (5(1X,D10.4),1X,A)
> lately reading sequential formatted external IO
> /data73/grid3-1.1.11/apps//atlas_app/atlas_rel/kitval/KitValid
> ation/JobTransforms/
> JobTransforms-11-00-42-09/share/csc.evgen.mcatnlo.trf:
> line 224: 20334 Aborted athena.py job.py 2>&1
> --------------------------------------------------------------
> --------------------
>
> I opened a Savannah bug #19105.
>
> 4. 7 jobs of the type csc11.005200.T1_McAtNlo_Jimmy.recotrig.v11000505
> failed at IU with "transExitCode=2: Athena core dump" due to:
> --------------------------------------------------------------
> -----------
> python:
> /N/Grid3/apps/atlas_app/atlas_rel/11.0.5/gcc-alt-3.2.3/lib/lib
> gcc_s.so.1:
> version `GCC_4.2.0' not found (required by
> /usr/lib/libstdc++.so.5) Wed Aug 16 05:20:38 EST 2006
> mv: cannot stat `ntuple.root': No such file or directory
> --------------------------------------------------------------
> -----------
>
> I created a RT ticket #430 at MWTier2.
>
> 5. 21 jobs of the type failed with "transExitCode=40: Athena
> crash - consult log file", they are all user jobs.
>
> 6. 1 job, csc11.005023.FJ4_pythia_jetjet.digit.v11004206._00653.job,
> failed with "transExitCode=50: Athena crash - consult log
> file" due to:
> --------------------------------------------------------------
> ---------
> ===> G4QGSMSplitableHadron - Fatal: Cannot sample parton
> densities under these constraints.
> G4HadronicProcess failed in ApplyYourself call for
> - Particle energy[GeV] = 16.702706
> - Material = Copper
> - Particle type = neutron
>
> *** G4Exception : 007
> issued by : G4HadronicProcess
> GeneralPostStepDoIt failed.
> *** Fatal Exception *** core dump ***
> --------------------------------------------------------------
> ----------
>
> Savannah bug #16730, a fix went into Release 12.
>
> 7. 18 jobs of the following type failed with "transExitCode=60:
> TRF_GBB_TIME - GriBB - output limit exceeded (time, memory, CPU)":
>
> testIdeal_06.005020.FJ1_pythia_jetjet.digit.v12000101
> testIdeal_06.005015.J6_pythia_jetjet.digit.v12000101
> testIdeal_06.005001.pythia_minbias.digit.v12000101
>
> Bug #18466 is closed, 48 hours limit was short for these jobs.
>
> 8. 74 jobs failed with "transExitCode=99: TRF_UNKNOWN -
> unknown transformation error"
>
> testIdeal_06.005015.J6_pythia_jetjet.digit.v12000101
> testIdeal_06.005145.PythiaZmumu.digit.v12000101
> testIdeal_06.005107.pythia_Wtauhad.digit.v12000101
>
> Bug #18349 is closed, fixed in tag LArG4EC-00-00-71 will go
> into 12.X.0 and 12.0.X.
>
> 9. 174 jobs of the type csc11.005200.T1_McAtNlo_Jimmy.digit.v11004205
> failed at UC_Teraport with "DQ2 staging input file failed"
> early morning:
>
> -------------- Log from
> /tmp/Panda_Pilot_15987_1155712210/dq2get.out ----- Getting
> POOL FileCatalog failed: cound not find the file in LRC!
> Could not get POOL FileCatalog!
> --------------------------------------------------------------
> -------------
>
> A RT ticket #427 was created by Tomasz.
>
> 10. Some test jobs were sent to OU_OSCER_ATLAS site to try to
> utilize a remote DQ2 server as opposed to a local NFS mounted
> one as requested by Karthik. It turned out that the pilots
> that were sent used the old version of DQ2ProdClient.py file.
> Xin was asked about how to use the new version of the file,
> DQ2ProdClient2.py, in this test.
>
> Regards,
> Nurcan.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Usatlas-prodsys-l mailing list
> [log in to unmask]
> http://lists.bnl.gov/mailman/listinfo/usatlas-prodsys-l
>
|
