 |
 |
> -----Original Message----- > From: Yang, Wei > Sent: Saturday, April 25, 2009 4:25 AM > To: Young, Charles C.; atlas-sccs-planning-l > Subject: Re: [Usatlas-prodsys-l] first HammerCloud jobs for US cloud > > Hi Charlie, > > I don't know the answer. HammerCloud is a new thing. I will > ask on Wed's US phone meeting. I do have some guesses: > > "mean prepare inputs time" should at least include copying > files from storage to batch nodes. This is likely the longest > part. It may also include querying local file catalogs for > file locations in storage. This should be pretty quick, in > seconds. I guess it doesn't include transferring data from > BNL to SLAC, as that will be a measurement of DDM > performance, and should be much longer than 48s. It also > shouldn't include time to access remote condDB. > > It is a little surprise that it took hours for other sites to > get data to their batch nodes. But if there are lots of input > files, that could be. For SLAC, we only copy non-ROOT files The logs indicate each job has about 150 files. Any idea how large each one is so we can do a reality check? > (like DBRelease.tar.gz) to batch nodes. > ROOT files are read directly from the storage. Again, shorter > MPIT doesn't mean our storage is better. > > There is no easy way to tell CPU utilization for a job > because there is no like between LSF IDs and Panda IDs. > > We report to US ATLAS a list of CPU types and numbers of > cores of each type. > US ATLAS calculates a weighted average based on their SI2K > (will use HEP-SPEC in the future), and use it to count the > CPU usage for each jobs. Thanks for the feedback. Are we sure about this point? The forwarded message has a pointer to http://gangarobot.cern.ch/st/test_253/, where we see the same 4 plots on page 3 of PowerPoint file. At this URL, they are explained as "CPU/Walltime is the CPU Percent Utilization", i.e. without cpufactor. It makes sense to monitor this ratio, but I don't understand the usage of a ratio utilization = cpuconsumption/cpufactor/(stoptime-starttime). that includes cpufactor. No idea what it would me. Maybe we can ask if the definition on page 2 is a typo. > > Wei Yang | [log in to unmask] | 650-926-3338(O) > > > > > > > From: "Young, Charles C." <[log in to unmask]> > > Date: Thu, 23 Apr 2009 23:48:50 -0700 > > To: Wei Yang <[log in to unmask]>, atlas-sccs-planning-l > > <[log in to unmask]> > > Cc: "Young, Charles C." <[log in to unmask]> > > Subject: RE: [Usatlas-prodsys-l] first HammerCloud jobs for US cloud > > > > Hi Wei, > > > > Thanks! Some questions. What is involved in the "mean > prepare inputs time" > > step? Is it copying input file from (local?) storage to > worker node? > > Is there some preparation of the input data beyond moving it around? > > > > Can we find CPU percent normalized to the execution step only? I.e. > > exclude file copy overhead. > > > > Don't understand definition on page 2. utilization = > > cpuconsumption/cpufactor/(stoptime-starttime). What is cpufactor? > >