 |
 |
Hi Niall, > Yes. The writes in particular can be very high rate and bursty (and the > sequence of the messages is very important, so its critical that they are > appended in the right order). Currently up to 125,000 messages to be > written a second, with another doubling in the next 12 months not entirely > out of the question. Each message is pretty small (fits in an ethernet > frame..). What are assuming your backing store will be? I say that because at 125,000 ops/sec is not at all easy to support with many of today's storage devices. Additionally, the intereference you're talking about from other streams (see below) poses additional burdens. You may also need to use more than one server here.The highest single-server op rate we've ever measured was about 90,000/sec (two CPU Sun v20z); using a DRAM storage model. I suspect that we would really have no problem, given the right hardware, with 125,000 ops/sec. > The reads are a mix of 'realtime' which are reasonable small as well, say > a few hundred kB, and 'historical' which may stream through a TB or more. > One critical aspect which your applications do not appear to have is that > the time between a message arriving and being available to data readers > must be as small as possible. Preferably milliseconds. The latency is not the problem here. So far we our tests have shown that xrootd will easily turn around a request, measured as a full roundtrip, in 60 microseconds (that's includes all 10Gb network and client overhead). Our measurements show that the theretical lower limit is probably on the order of 20 microseconds assuming that you're using a 2.8GHZ Intel processor and infiniband. We could probably do better with some tinkering. Also remember that single stream latency does not necessarily correlate with ops/sec.We have yet to figure out what that correlation is for xrootd but we do know that it is a precondition for a high transaction rate. Our focus on latency comes from the PetaCache project where we are trying to support thousands of parallel random read requests. Now, what is the problem is the "other" non-high transaction rate work that would be going on at the same time. We've found that mixing such workloads on a single server or device hurts one or the other and many times both. I don't think you can expect a high transaction rate if the data holder has to also support "bulk" requests. Andy