 |
 |
Hi Adrian, I was the one who was performing these speed tests, so let me give a little bit of detail about them. We had 6 file, each of size 1GB. These files were placed one per tray on each of 6 trays of 15 2-TB disks in RAID-6. The data server in question has 24GB of RAM, and a 10Gb ethernet link. The test was performed from 100 worker nodes, which all attempted to read all six files in parallel. As a note, it's unlikely that this was bottlenecked or affected by the RAID cards because it's pretty clear that all 6 files would easily fit in the kernel page cache. So this is mostly a test of page-cache->xrootd->NIC throughput rather than RAID throughput. I hope to do some more comprehensive testing which will include the RAID controller in the mix, but the interesting result here is simply that the softirqd symptom that Charles mentions was in fact our bottleneck, and when that was removed we were able to move our data at line speed. -Aaron On Apr 4, 2011, at 10:10 AM, Adrian Sevcenco wrote: > On 04/04/2011 05:52 PM, Charles G Waldman wrote: >> >> > Are any of xrootd team aware of >> > http://twiki.mwt2.org/bin/view/ITB/UltraLightKernel ? are there any (up >> > to date) recommendations based on this guys work? >> >> Hi - > Hi! > >> I'm involved with both xrootd and the configuration of the Ultralight >> kernel. > great! you guys do an superb job (as far i seen in graphs :) ) > >> The major difference between the UltraLight kernel and the stock SL >> kernel is that the SL kernel is configured with preemption enabled, >> which is great for a user desktop, but not for a high-throughput >> data server. > and is this the only difference? > >> We have found that on our test server (16-core server Dell 2950 with 6 >> RAID6 shelves) with the SL kernel, under load, the "ksoftirqd" daemon > so the speed benchmark is done accessing all 6 raid6 arrays or just one raid? > >> takes up 100% CPU (this is a kernel thread involved in preemption), >> and the xrootd process is only able to deliver data at about 500MB/sec >> (about 50% of theoretical max on a on a 10Gb link). >> >> With the UL kernel, the "ksoftirq" CPU usage goes away completely, and >> xrootd is easily able to deliver data at wire speed (~1000MB/sc) > great news!!! > > Thanks a lot! > Adrian > >> >> - Charles >