Another thing that has been consistently a problem with the cluster is the seeming inability to kill a job. I hit Cancel/Stop on the GUI and waited for about 10 minutes for the process to complete. This did nothing, so I hit Ctrl-C, which got me this:
Enter A/a to switch asynchronous, S/s to stop, Q/q to quit, any other key to continue: s
Info in <TSignalHandler::Notify>: Processing interrupt signal ... s
Enter A/a to switch asynchronous, S/s to stop, Q/q to quit, any other key to continue: s
Info in <TSignalHandler::Notify>: Processing interrupt signal ... s
Info in <TMonitor::Select>: *** interrupt occured ***
Enter A/a to switch asynchronous, S/s to stop, Q/q to quit, any other key to continue: s
Info in <TSignalHandler::Notify>: Processing interrupt signal ... s
Info in <TMonitor::Select>: *** interrupt occured ***
Selecting q for quit doesn't seem to do anything either. Eventually I had to log into atlint01 again and kill the root process manually. On a subsequent attempt, I was able to reconnect successfully, but my previous session was still running (or crashed) and it just reconnected to that (stalled) session. I'm trying to kill that session again now so I can run a longer test.
Short version: there seems to be an issue with killing jobs cleanly which seems to leave the cluster in an unusable state (at least until some internal timeout completes?), but it does seem to be running (from the T2 storage at the moment, longer T2 vs. cluster storage performance test in the works assuming this session ever dies) using all workers provided the shared library is pre-compiled. This is something I saw in local sessions and back in May, so that it is not really surprising that it's an issue with the cluster too considering my code to make the package hasn't changed much. Figuring out on-the-fly compilation on the workers would be nice down the road. Getting the ability to kill jobs cleanly now is necessary though.
-Bart
Yang, Wei wrote:
Hi Bart,
I made another attempt. Here is what I used to start at client side (assuming bash) on a rhel5-64 machine
. /afs/slac/g/atlas/packages/gcc432/setup.sh
export ROOTSYS=/afs/slac.stanford.edu/g/atlas/packages/root/root5.26.00b-slc5_amd64-gcc43
export PATH=${PATH}:/$ROOTSYS/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ROOTSYS/lib
$ROOTSYS/bin/root
It seems I was able to load a .par file. Can you give it a try? Also, remember on atlint01, if you copy a file to /xrootd/proof/bcbutler, you should TDset::Add("root://boer0123//atlas/proof/bcbulter/..."). However, I found that reading from T2 storage seems to be faster than reading from the disks in the proof cluster (without localizer).
regards,
Wei Yang |
[log in to unmask]
| 650-926-3338(O)
On Apr 29, 2010, at 3:25 PM, Bart Butler wrote:
First thing first, I think I killed your cluster. The xrootd mount is no longer readable from atlint01 and I can't submit PROOF jobs to it anymore. This happened after killing my client root session manually after a massively-screwed up job.
Secondly, I am have a hell of a time compiling my shared library correctly. Which version of ROOT is the cluster running? If I'm not running the exact same root version and gcc version as every worker node, I can't make binaries (which is what Booker did with his test package it seems. I do it too when I run PROOF-Lite). And if I can't make binaries, I have to submit source packages. This should be fine but it's never worked well for me. My first theory was that because the packages are kept in a common place on xrootd in my user space, the compilation errors I was getting from some workers were because all 32 (I was never able to connect to 4 of the 36 workers) tried to compile the package at the same time in the same place. Running on a single worker worked fine (but of course was slow). I don't think this compilation issue was the whole story though, because if the single worker thing worked, the next time all workers should have been able to load the compiled version with
out problems assuming they are all running the same version of ROOT, and they crashed and burned just as badly that time. That's when the cluster itself crashed.
Another thing was that making TDSets from the Tier 2 xrootd storage worked fine, but when I tried using the same files I had copied to the cluster xrootd storage it couldn't find them for some reason.
My log files should be in /xrootd/proof/bcbutler if you guys get the cluster working again.
-Bart
Yang, Wei wrote:
Hi Bart, David,
any news on this?
regards,
Wei Yang |
[log in to unmask]
| 650-926-3338(O)
On Apr 21, 2010, at 12:03 PM, Bart Butler wrote:
I'll try to run a few jobs tonight and see what happens.
-Bart
Yang, Wei wrote:
[add Andy Hass ...]
Hi David, Booker,
I mounted the xrootd space of the proof cluster at /xrootd/proof on atlint01. It looks like we have ~1.8TB total on the cluster. So something ~ 1TB should work.
The cluster should be able to access T2 storage if your provide the URL of those root file to process. But the whole idea of using proof is to avoid network traffic as much as possible. As we are still validation the functions, it would be good to try both. Or if you put half of the data on proof cluster, and leave the other half on T2 storage (no NFS please).
The proof master node is boer0123. If you copy files to the cluster, the xroot URL is root://boer0123//atlas/proof (I suggest you to create a fizisist sub-dir).
Booker, it looks like proof also leaves some file in the cluster. How would you suggest to manage the space, by user, by group, or something else?
regards,
Wei Yang |
[log in to unmask]
| 650-926-3338(O)
On Apr 21, 2010, at 8:40 AM, David W. Miller wrote:
Hi Booker and Wei,
I have a few questions: from what machine do we launch the jobs? Any machine at SLAC, but specifying the URI correctly? Also, if the data are on atlasuserdisk or usr in /xrootd/atlas/ is that sufficient?
Thanks,
David
On Apr 21, 2010, at 17:36 PM, Ariel Schwartzman wrote:
From: Booker Bense <[log in to unmask]>
Date: April 21, 2010 16:09:51 PM GMT+02:00
To: "Schwartzman, Ariel G."
<[log in to unmask]>
Cc: "Yang, Wei"
<[log in to unmask]>
Subject: Re: Proof cluster ready for testing
On Wed, 21 Apr 2010, Ariel Schwartzman wrote:
Hi Booker,
I cannot access this machine remotely:
ssh -Y boer0123.slac.stanford.edu
ssh: connect to host boer0123.slac.stanford.edu port 22: Operation timed out
It's on the slac internal network, you'll need to login to a slac
machine and run root programs from there. You shouldn't need
login access to the master node.
_ Booker C. Bense
==========================================
David W. Miller
------------------------------------------
SLAC
Stanford University
Department of Physics
SLAC Info: Building 84, B-156. Tel: +1.650.926.3730
CERN Info: Building 01, 1-041. Tel: +41.76.487.2484
EMAIL:
[log in to unmask]
HOMEPAGE:
http://cern.ch/David.W.Miller
==========================================
