Hi Daniele,

do you have an example of a log file for these jobs ? I do not know
exactly what servers these disks have been installed on, but we
noticed in E158, where most of the data were sitting on one
(relatively slow) server, jobs were limited by I/O throughput to about
2 MB/sec. This limit comes from the random access pattern that split
ROOT trees provide. If your job is sufficiently fast, you can saturate
I/O limit quite quickly -- with 2-3 jobs. If you submit too many jobs
(tens or even hundreds), the server will thrash to the point that the
clients will receive NFS timeouts. ROOT usually does not like that --
you may see error messages in the log file about files not found (when
the files are actually on disk), or about problems uncompressing
branches. These are usually more severe on Linux clients, where the
NFS client implementation is not very robust.. 

There are several ways to cope with this problem:

1) Submit fewer jobs at one time. I would not submit more than 10
   I/O-limited jobs in parallel. 
2) Place your data on different servers. That means, different sulky
   servers is best. Even if you are on the same sulky server but split
   your data onto different partitions, you still get the benefit of
   parallelizing disk access
3) Re-write your jobs to first copy your data onto a local disk on the
   batch worker (for instance, /tmp), then run on the local copy, then
   delete the local copy. The benefit of that is that the cp command
   will access the file in direct-access mode (with 10-20 MB/sec
   throughput, depending on the network interface throughput). 
4) Make your ntuples non-split (very highly recommended). This usually
   increases the throughput by a factor of 10-20. If your typical job
   reads most of the branches of the tree, making tree split makes no
   sense. Non-split trees provide direct access to disk, which is much
   more optimal. 

							Yury


On Thu, Oct 31, 2002 at 09:26:08AM -0800, Daniele del Re wrote:
> 
> Hi all,
> 
>  in the last two days I tried to run on data and MC on the new disk AWG18.
> No way. I got problems in the 80% of the jobs. Someone crashed, most of
> them have did not read a large number of root files (actually there).
> 
>  This problem seems to be worse than ever. Do we have to contact
> computing people about this?
> 
>  Daniele
> 
>