Hi Yury,
one example is
~daniele/scra/newchains_1030/data-2
and the tipical message is
Error in <TFile::TFile>: file /nfs/farm/babar/AWG18/ISL/sx-080702/data/2000/output/outputdir/AlleEvents_2000_on-1095.root does not exist
on AWG8 this pathology happened just few times when there were >~300 jobs
reading the same disk if I remember correctly.
Do you know which is the difference between AWG8 and AWG18?
My proposal is to split things on different disks, if possible.
Thanks a lot,
Daniele
On Thu, 31 Oct 2002, Yury G. Kolomensky wrote:
> Hi Daniele,
>
> do you have an example of a log file for these jobs ? I do not know
> exactly what servers these disks have been installed on, but we
> noticed in E158, where most of the data were sitting on one
> (relatively slow) server, jobs were limited by I/O throughput to about
> 2 MB/sec. This limit comes from the random access pattern that split
> ROOT trees provide. If your job is sufficiently fast, you can saturate
> I/O limit quite quickly -- with 2-3 jobs. If you submit too many jobs
> (tens or even hundreds), the server will thrash to the point that the
> clients will receive NFS timeouts. ROOT usually does not like that --
> you may see error messages in the log file about files not found (when
> the files are actually on disk), or about problems uncompressing
> branches. These are usually more severe on Linux clients, where the
> NFS client implementation is not very robust..
>
> There are several ways to cope with this problem:
>
> 1) Submit fewer jobs at one time. I would not submit more than 10
> I/O-limited jobs in parallel.
> 2) Place your data on different servers. That means, different sulky
> servers is best. Even if you are on the same sulky server but split
> your data onto different partitions, you still get the benefit of
> parallelizing disk access
> 3) Re-write your jobs to first copy your data onto a local disk on the
> batch worker (for instance, /tmp), then run on the local copy, then
> delete the local copy. The benefit of that is that the cp command
> will access the file in direct-access mode (with 10-20 MB/sec
> throughput, depending on the network interface throughput).
> 4) Make your ntuples non-split (very highly recommended). This usually
> increases the throughput by a factor of 10-20. If your typical job
> reads most of the branches of the tree, making tree split makes no
> sense. Non-split trees provide direct access to disk, which is much
> more optimal.
>
> Yury
>
>
> On Thu, Oct 31, 2002 at 09:26:08AM -0800, Daniele del Re wrote:
> >
> > Hi all,
> >
> > in the last two days I tried to run on data and MC on the new disk AWG18.
> > No way. I got problems in the 80% of the jobs. Someone crashed, most of
> > them have did not read a large number of root files (actually there).
> >
> > This problem seems to be worse than ever. Do we have to contact
> > computing people about this?
> >
> > Daniele
> >
> >
>
>
|