 |
 |
Hi all, very interesting discussion. What I still do not know or understand is whether your data model (messages+store) is forcefully based on the concept of "file" or not. What I intend is that if you feel that you need something different, you still can use the xrootd architecture by rewriting one or more plugins in order to match your needs. That's what has been done e.g. for PROOF at CERN. From what you say your app looks like a sequential logger, but I missed an estimation of the number of concurrent writers. For an optimization of the read case, instead, maybe you can do a very nice job after having analyzed the access patterns. Unfortunately I haven't submitted yet to the website (through Pete) the last papers/presentations dealing with the access optimizations. I will do it asap. Fabrizio Niall Dalton wrote: > Hi Peter, > > Thanks for the response. > >> This may not be what you are looking for, but here is an example >> note on high rate I/O disk server benchmarks > > Any such information is useful, so thanks for the link. > >> 25GB isn't actually a lot over 8 hours > > :-) I'm obviously asking in the right place then. 26GB a day is > considered rather large for the groups in question. > >> sounds like you are more interested in very high transaction rates for >> very small reads/writes, is that correct? > > Yes. The writes in particular can be very high rate and bursty (and the > sequence of the messages is very important, so its critical that they > are appended in the right order). Currently up to 125,000 messages to be > written a second, with another doubling in the next 12 months not > entirely out of the question. Each message is pretty small (fits in an > ethernet frame..). > > The reads are a mix of 'realtime' which are reasonable small as well, > say a few hundred kB, and 'historical' which may stream through a TB or > more. One critical aspect which your applications do not appear to have > is that the time between a message arriving and being available to data > readers must be as small as possible. Preferably milliseconds. > >> (Could you tell us a bit more about your application? ;-) > > Very little I'm afraid. The data are stock market 'ticks' - quotes for > equity options for example, or foreign exchange. Some trading > applications need to respond within a second to messages, and can take > profitably take advantage of sophisticated data analyses. > > Other applications are more batch in nature, but these are easier to > scale with existing techniques. > > niall