ATLAS SCCS Planning 09May2007 ----------------------------- 9am, SCCS Conf Rm A, to call in +1 510 665 5437, press 1, 3935# Present: Charlie, Len, Wei, Renata, Stephen Agenda: 1. DQ2 Status/Web Proxy Have the new SLACXRD working, had 90 jobs finish successfully yesterday. Each server has 8TB. There is another 9TB on each server for local users. 2. Tier-2 Hardware Should make the 9TB on each server available via xrootd. One advantage of moving is the larger space available. It isn't easy to do an "ls" on the xrootd though, so perhaps we should also make these available via NFS. Could also have another DQ2 instance to have a database of that area. It seems that it is better to use DQ2 rather than dq2_get to bring files to SLAC. One issue with that xrootd is that there only one owner of all the files. Can use the automatic backup via xrootd so that disk space isn't limited due to old unused data hanging around. If we had two instances of DQ2 it could bring two copies of the same file. For the moment make it available via xrootd and see if we should make it available via NFS also. Lance is using the third box for doing benchmark and reliability tests. Need to also think about mirroring the system disk. This would take up 2 500GB disks. Could make the other 800GB available somehow, but this would effectively be lost. Probably not worth mirroring. Not heard anything recently about the CPU, everything was on track at last update. 3. xrootd/srm No news. 4. AFS Problems causing jobs to intermittently fail From Renata about BaBar problems; "First some AFS background. An AFS file server keeps track of client requests with callbacks. A callback is a promise by the file server to the tell the client when a change is made to any of the data being delivered. This can have an impact on server performance in the following ways: 1. The performance of an AFS server can become seriously impaired when many clients are all accessing the same read-write file/directory and that file/directory is being updated frequently. Every-time an update is made, the file server needs to notify each client. So, a large number of clients can be a problem even if the number of updates is relatively small. 2. The problem outlined above can be further exacerbated if a large number of requests for status are made on the file/directory as soon as the callbacks are broken. A broken callback will tell the client to refetch information, so the larger the number of machines, the larger the number of status requests that will occur as a result of the broken callback. And then any additional status requests that may be going on will cause further grief. The way to avoid callback problems is to avoid writing to the same file/directory in AFS from many clients. The recommended procedure in batch is to write locally and copy once to AFS at the end of the job. The problems that we saw with BaBar: First I should say that the problems we saw with BaBar came after they started increasing the number of jobs being run as part of their skimming. Before that, the problems were still there, but at a low enough level that they didn't have the same impact. 1. There was a problem with our TRS utility that was causing multiple updates to a file in one of their AFS directories. This was causing the problem described above. We have since changed the TRS utility to avoid making that update. 2. The BaBar folks were launching 1000s of batch jobs at once which were accessing the file(s) on one server in such a way that it caused a plunge in availability. They have since changed the way they run by keeping the level of batch jobs up so that 1000s don't hit all at the same time, but are spread out. We are still trying to figure out what the jobs are doing at startup that cause the problem (writing to AFS?), but the bypass has been working. I have our AFS support people looking into it. 3. The BaBar folks also fixed a problem in their code that was launching 10s of 1000s of 1 minute batch jobs. This was putting a heavy load on the batch system because it had to spend much/all of its time scheduling, in addition to the impact on AFS. 4. The BaBar code does huge numbers of accesses to files under /afs/slac/g/babar. They suspect that their tcl files are part of the problem and they are going to move those files to readonly volumes. This will spread the load across multiple machines. Unfortunately the BaBar group space has grown over time so that setting it up to be readonly now is a daunting task. At the moment they have a parallel readonly volume that they will be using for the tcl space. A little AFS background on readonly volumes....the readonly path through AFS requires that all volumes (mountpoints) along the way be readonly. So, in the case of the atlas volume /afs/slac/g/atlas/AtlasSimulation for example, /afs/slac/g/atlas would have to be set up with readonlies in order for AtlasSimulation to be set up with readonlies. So if you think some of your code would benefit from having the load spread across multiple fileservers in readonly volumes, it would be best to set up time to switch /afs/slac/g/atlas to be readonly now, before things get anymore complicated." And from Len about read-only volumes; "I thought I should add some comments about why we have not pushed the use of read-only clones more heavily. The AFS command to update read-only clones from the read-write volume is 'vos release'. This is a privileged AFS command and the privilege is global, that is, it is not attached to particular volumes: if you've got this privilege, you can vos release any cloned volume in the AFS cell. (IIRC, the same privilege allows you to run opther privileged vos commands.) We have a SLAC-written wrapper, 'vos_release', for the native AFS command that allows AFS package maintainers to do vos releases for the volumes in their packages. The authorization scheme for this wrapper makes use of our naming conventions for package volumes and for the AFS groups in package space. However, AFS group space is much less regular than package space, and our simple wrapper would scale well if we tried to provide fine-grained authorization for vos releases in group space. What we are currently looking into for BaBar is to define a single AFS group whose members would be able to do a vos release for any cloned BaBar volume (all such volume names begin with 'g.bbr'). We have also asked that BaBar keep the number of people in the AFS group small (e.g., 5-10). With this sort of scheme, you probably only want to clone volumes that change infrequently. This, coupled with the need to have clones on all parent volumes, implies constraints on how the space is organized." With ATLAS have seen some files not be able to be read. Have expected that a job would wait a long time but not think the file doesn't exist. Have seem problems like this but not been able to track them down. The ATLAS problems did seem to correlate with when BaBar had problems. Could make the top level read-only. This will mean separating out some things from that volume as it should be small. Need to build in some sort of authorisation scheme to allow ATLAS folk do the "vos release" on ATLAS space. Will wrap the command that communicates with a privileged server that does the actual "vos release". Not talked to Alf yet about this, need to discuss with him to see what schemes are reasonable to implement. One issue might be that the ATLAS release remembers where it is installed so it might remember the read-write path instead of the read-only one. Is having the NFS space mapped to users via AFS could a problem? Don't believe so but there is hte issue with NFS opening and closing files for each access might cause some worry. Could replicate the top level volume three times. Will check if we're still seeing problems running ATLAS software. Will report it to unix-admin next time we see it. 5. AOB None. Action Items: ------------- 070509 Stephen Split up top-level AFS volume, requests to unix-admin 070502 Stephen Email Gordon about his action item 070509 Done. 070502 Stephen Arrange meeting about ATLAS TAG data on PetaCache 070509 Not done yet. 070502 Wei Check CA certificate update mechanism 070509 Not done yet but believe VDT is the right way. 070321 Gordon Discuss perception of SLAC Tier-2 with external folk. 070404 no info 070411 no info 070509 No info.