 |
 |
Sorry Alessio, I am not sure I understand your proposal: you suggest to produce the reduced root files and make chains over them? But these root file are only 3 times smaller than the default ones, how can you claim we will need more than ten times less jobs? thanks ric On Thu, 31 Oct 2002, Alessio Sarti wrote: > Hi all, > I propose another workaround! > > I've produced the chains. > I can split them and run the skim job that produces in lees thatn 2 hours > 6-10 job contaiing ALL the generic / data/ cocktail info in such a way > that we donot longer rely on hundreds of job running..... > > What do you think about that? > Alessio > > ______________________________________________________ > Alessio Sarti Universita' & I.N.F.N. Ferrara > tel +39-0532-781928 Ferrara > roma +39-06-49914338 > SLAC +001-650-926-2972 > > "... e a un Dio 'fatti il culo' non credere mai..." > (F. De Andre') > > "He was turning over in his mind an intresting new concept in > Thau-dimensional physics which unified time, space, magnetism, gravity > and, for some reason, broccoli". (T. Pratchett: "Pyramids") > > On Thu, 31 Oct 2002, Daniele del Re wrote: > > > > > Hi Yury, > > > > one example is > > > > ~daniele/scra/newchains_1030/data-2 > > > > and the tipical message is > > > > Error in <TFile::TFile>: file /nfs/farm/babar/AWG18/ISL/sx-080702/data/2000/output/outputdir/AlleEvents_2000_on-1095.root does not exist > > > > on AWG8 this pathology happened just few times when there were >~300 jobs > > reading the same disk if I remember correctly. > > > > Do you know which is the difference between AWG8 and AWG18? > > > > My proposal is to split things on different disks, if possible. > > > > Thanks a lot, > > > > Daniele > > > > On Thu, 31 Oct 2002, Yury G. Kolomensky wrote: > > > > > Hi Daniele, > > > > > > do you have an example of a log file for these jobs ? I do not know > > > exactly what servers these disks have been installed on, but we > > > noticed in E158, where most of the data were sitting on one > > > (relatively slow) server, jobs were limited by I/O throughput to about > > > 2 MB/sec. This limit comes from the random access pattern that split > > > ROOT trees provide. If your job is sufficiently fast, you can saturate > > > I/O limit quite quickly -- with 2-3 jobs. If you submit too many jobs > > > (tens or even hundreds), the server will thrash to the point that the > > > clients will receive NFS timeouts. ROOT usually does not like that -- > > > you may see error messages in the log file about files not found (when > > > the files are actually on disk), or about problems uncompressing > > > branches. These are usually more severe on Linux clients, where the > > > NFS client implementation is not very robust.. > > > > > > There are several ways to cope with this problem: > > > > > > 1) Submit fewer jobs at one time. I would not submit more than 10 > > > I/O-limited jobs in parallel. > > > 2) Place your data on different servers. That means, different sulky > > > servers is best. Even if you are on the same sulky server but split > > > your data onto different partitions, you still get the benefit of > > > parallelizing disk access > > > 3) Re-write your jobs to first copy your data onto a local disk on the > > > batch worker (for instance, /tmp), then run on the local copy, then > > > delete the local copy. The benefit of that is that the cp command > > > will access the file in direct-access mode (with 10-20 MB/sec > > > throughput, depending on the network interface throughput). > > > 4) Make your ntuples non-split (very highly recommended). This usually > > > increases the throughput by a factor of 10-20. If your typical job > > > reads most of the branches of the tree, making tree split makes no > > > sense. Non-split trees provide direct access to disk, which is much > > > more optimal. > > > > > > Yury > > > > > > > > > On Thu, Oct 31, 2002 at 09:26:08AM -0800, Daniele del Re wrote: > > > > > > > > Hi all, > > > > > > > > in the last two days I tried to run on data and MC on the new disk AWG18. > > > > No way. I got problems in the 80% of the jobs. Someone crashed, most of > > > > them have did not read a large number of root files (actually there). > > > > > > > > This problem seems to be worse than ever. Do we have to contact > > > > computing people about this? > > > > > > > > Daniele > > > > > > > > > > > > > > > > > > > > > >