Hi,
We finished our initial testing of 8-core machines at DZERO for the Level-3 trigger/DAQ (these are configured as a large farm running a software-only trigger). Unfortunately, before we could run careful memory usage tests we had to return the tests. The node had 8 cores and 8 gigs of memory (x64, obviously).
We did have to make one change to the linux kernel (easy): /proc/sys/kernel/msgmni -- up that to 64 message queues. But I doubt normal usage will have to deal with that -- we use these message queue's to coordinate the movement of data through the system. We don't have exactly 8 independent executables. Rather we have an event builder in each node which sends fully built events to one of 8 trigger executables. These message queues do the coordination.
As far as the simple memory pressure tests we did do, with 7 of the images running there didn't seem to be much of a problem. Certainly it was not the bottle neck.
Sorry this isn't more helpful. But at least for the DØ Trigger we've not hit any limits.
Cheers,
Gordon.
P.S. I remember the VAX where we talked about never _ever_ using up all the address space that 32 bit machine had. Ha!
> -----Original Message-----
> From: [log in to unmask] [mailto:owner-
> [log in to unmask]] On Behalf Of Stephen J. Gowdy
> Sent: Wednesday, August 15, 2007 7:07 PM
> To: ATLAS SCCS Planning
> Subject: Minutes of ATLAS/SCCS Planning Meeting 15th August 2007
>
> ATLAS SCCS Planning 15Aug2007
> -----------------------------
>
> 9am, SCCS Conf Rm A, to call in +1 510 665 5437, press 1, 3935#
>
> Present: Wei, Stephen, Len, Richard, Chuck, Booker
>
> Agenda:
>
> 1. DQ2 Status/Web Proxy
>
> BNL is reallocating machines just now so have shutdown. Also
> problems at CERN with central DQ2 services. Things at SLAC are
> running fine.
>
> 2. Tier-2 Hardware
>
> Every site should have the new hardware up and running by 1st
> April. Several sites had different kinds of estimates, SLAC for
> instance assumed the 2008 commitments were needed at the end of
> 2007. Thing that JimS took 20% off the contributions suggested from
> the sites.
>
> The longer we wait the better value we can get but it will also
> mean we risk loosing the rack space where they would go. Right now
> PPA is deciding who uses what space. We know just now where it
> would go but in six months. Having had a bit of trouble this year
> with having the money but no equipment thing we should think about
> buying earlier than later.
>
> There is the water cooled racks and some space on the second floor
> (need to worry a little about weight). After that it isn't clear
> where we'll have space. It might all be occupied by PII, BSD,
> etc. It is first come first served with the option that anyone can
> go to the management to get that changed. It will dawn on everyone
> that we run out of space at the end of the year more folk will
> (need to) go that route.
>
> The BaBar purchase needs to be up and running by the 1st of the
> year. This needs a new Black Box. The purchase is being determined
> based on the total power that can be put in it. Not likely to be
> the fastest and hottest CPUs you can get. Might get more CPU/$$
> going down a notch or two for the power limit of Black Box. In the
> water cooled racks you need to get the most expensive CPU to get
> the most out of them. It is something like $500k for eight racks
> (of around 40U). This will take around $4M in computing
> power. Normally to a outside contract you add 60% without CEF
> actually doing anything, with about $350k going to the
> contractor. The Black Box comes out something like $3k rack unit in
> black box, but you are limited by power.
>
> The infrastructure is getting more expensive than the machines. A
> white paper from Stanford shows this and also the electrical bill
> getting more expensive than the hardware in the future. Have also
> had in the past that people dominated 2:1 in the past but this
> trend is reversing.
>
> Thinking of building some terraces into the hillside around
> IR8. Try to do it as low cost but well planned as possible. Looking
> out three years ahead. This will provide facilities to SLAC and
> Stanford. Need to somehow survive till then, perhaps more water
> cooled racks but the chilled water in Building 50 is also at
> capacity. Black Boxes also can't be added in the same place as need
> separate power and cooling provided. The industry might learn as
> people can buy less hardware as they are paying more for
> infrastructure that they need to make cooler stuff.
>
> So our purchase should probably be on the same timescale as the
> BaBar one. They are in a hurry though. We should keep open that we
> use the same purchasing and evaluation effort. Perhaps the ATLAS
> stuff gets the same hardware with faster CPUs in it. So will
> probably tie these together.
>
> 3. AOB
>
> - SLAC ATLAS Group Allocation
>
> Came to Richard's knowledge that currently the SLAC ATLAS group
> has a special fairshare (the group in question isn't actually for
> the SLAC ATLAS group, but for any ATLAS users wanting to use the
> Tier-2 in batch mode) on the Tier-2. Can not run a Tier-2 and
> give your local users special access. There should be lab funded
> machines for general use for PPA employees. Will put a large
> fraction of Black Box #1 into this. Then all groups will get
> their what they expect.
>
> Need to try to keep separate "local" users and general ATLAS. The
> "local" is the informal consortium of universities that supports
> the Tier-2. Need to be able to support analysis at the Tier-2 by
> giving them enough cycles etc. For local analysis activities can
> only let in folk in the Western Tier-2 Consortium. Many people
> are trying to do their own thing at their institution so the load
> may not be as high as expected.
>
> As a local person you could use the Tier-2 as a general ATLAS
> person or the Tier-3 as a local user. The same issue is coming up
> at CERN for local analysis here.
>
> Technically it isn't difficult to setup another LSF group. Not
> quite sure how to separate disk space usage though. Lab
> management has not been asked about an AllUsers disk pool. If
> there will be heavy use need some disk space funded as cannot use
> the Tier-2 disk space for local usage only. It is thought that
> the 20% reduction by JimS was for US usage. Trying to serve a
> diffuse community with a storage area isn't easy, best to have
> well identified set of users.
>
> Something that works for a production activity with one master
> won't necessarily work in a more chaotic usage. BaBar is looking
> at using the local disk as a temporary storage area for skimming,
> which could be there for many hours. Could end up having more
> storage on the nodes due to the more-or-less minimum size disks
> coming with machines these days.
>
> - xrootd
>
> Some issues with redirector and ATLAS software, not sure where
> the problem is.
>
> Wei things there may also be some problems with 64 bit but there
> are some other issues Andy should address.
>
> Action Items:
> -------------
>
> 070815 Wei Thing about how we maintain lists of local people etc
>
> 070801 Stephen See when new Tier-2 hardware is needed
> 070815 Received answers independently.
>
> 070725 Stephen Try to test eval01
> 070801 Didn't have access when attempted, Booked fixed that.
> Problem with ATLAS software (hopefully trivial).
> 070815 Installed new software to get around the problem. Not
> tested yet. Have order 128 cousins for it, have bought
> the machine.
>
> 070711 Stephen Find out about benchmarks for CPUs for next
> purchase
> 070718 Not done yet.
> 070801 Extracted data from our Production Database, need to
> analyse it still.
> 070815 Not done yet.
|
