 |
 |
Hi all, I propose another workaround! I've produced the chains. I can split them and run the skim job that produces in lees thatn 2 hours 6-10 job contaiing ALL the generic / data/ cocktail info in such a way that we donot longer rely on hundreds of job running..... What do you think about that? Alessio ______________________________________________________ Alessio Sarti Universita' & I.N.F.N. Ferrara tel +39-0532-781928 Ferrara roma +39-06-49914338 SLAC +001-650-926-2972 "... e a un Dio 'fatti il culo' non credere mai..." (F. De Andre') "He was turning over in his mind an intresting new concept in Thau-dimensional physics which unified time, space, magnetism, gravity and, for some reason, broccoli". (T. Pratchett: "Pyramids") On Thu, 31 Oct 2002, Daniele del Re wrote: > > Hi Yury, > > one example is > > ~daniele/scra/newchains_1030/data-2 > > and the tipical message is > > Error in <TFile::TFile>: file /nfs/farm/babar/AWG18/ISL/sx-080702/data/2000/output/outputdir/AlleEvents_2000_on-1095.root does not exist > > on AWG8 this pathology happened just few times when there were >~300 jobs > reading the same disk if I remember correctly. > > Do you know which is the difference between AWG8 and AWG18? > > My proposal is to split things on different disks, if possible. > > Thanks a lot, > > Daniele > > On Thu, 31 Oct 2002, Yury G. Kolomensky wrote: > > > Hi Daniele, > > > > do you have an example of a log file for these jobs ? I do not know > > exactly what servers these disks have been installed on, but we > > noticed in E158, where most of the data were sitting on one > > (relatively slow) server, jobs were limited by I/O throughput to about > > 2 MB/sec. This limit comes from the random access pattern that split > > ROOT trees provide. If your job is sufficiently fast, you can saturate > > I/O limit quite quickly -- with 2-3 jobs. If you submit too many jobs > > (tens or even hundreds), the server will thrash to the point that the > > clients will receive NFS timeouts. ROOT usually does not like that -- > > you may see error messages in the log file about files not found (when > > the files are actually on disk), or about problems uncompressing > > branches. These are usually more severe on Linux clients, where the > > NFS client implementation is not very robust.. > > > > There are several ways to cope with this problem: > > > > 1) Submit fewer jobs at one time. I would not submit more than 10 > > I/O-limited jobs in parallel. > > 2) Place your data on different servers. That means, different sulky > > servers is best. Even if you are on the same sulky server but split > > your data onto different partitions, you still get the benefit of > > parallelizing disk access > > 3) Re-write your jobs to first copy your data onto a local disk on the > > batch worker (for instance, /tmp), then run on the local copy, then > > delete the local copy. The benefit of that is that the cp command > > will access the file in direct-access mode (with 10-20 MB/sec > > throughput, depending on the network interface throughput). > > 4) Make your ntuples non-split (very highly recommended). This usually > > increases the throughput by a factor of 10-20. If your typical job > > reads most of the branches of the tree, making tree split makes no > > sense. Non-split trees provide direct access to disk, which is much > > more optimal. > > > > Yury > > > > > > On Thu, Oct 31, 2002 at 09:26:08AM -0800, Daniele del Re wrote: > > > > > > Hi all, > > > > > > in the last two days I tried to run on data and MC on the new disk AWG18. > > > No way. I got problems in the 80% of the jobs. Someone crashed, most of > > > them have did not read a large number of root files (actually there). > > > > > > This problem seems to be worse than ever. Do we have to contact > > > computing people about this? > > > > > > Daniele > > > > > > > > > > > > >