Print

---

 > Are any of xrootd team aware of 
 > http://twiki.mwt2.org/bin/view/ITB/UltraLightKernel ? are there any (up 
 > to date) recommendations based on this guys work?

Hi - 

 I'm involved with both xrootd and the configuration of the Ultralight
kernel.

 The major difference between the UltraLight kernel and the stock SL
kernel is that the SL kernel is configured with preemption enabled,
which is great for a user desktop, but not for a high-throughput
data server.

 According to the kernel configuration docs:

 CONFIG_PREEMPT_NONE:

   This is the traditional Linux preemption model, geared towards
   throughput. It will still provide good latencies most of the
   time, but there are no guarantees and occasional longer delays
   are possible.

   Select this option if you are building a kernel for a server or
   scientific/computation system, or if you want to maximize the raw
   processing power of the kernel, irrespective of scheduling
   latencies.

This is what Ultralight uses.


The SL/Redhat kernel is configured with 

 CONFIG_PREEMPT: 

   This option reduces the latency of the kernel by making all kernel
   code (that is not executing in a critical section) preemptible.
   This allows reaction to interactive events by permitting a low
   priority process to be preempted involuntarily even if it is in
   kernel mode executing a system call and would otherwise not be
   about to reach a natural preemption point.  This allows
   applications to run more 'smoothly' even when the system is under
   load, at the cost of slightly lower throughput and a slight runtime
   overhead to kernel code.

   Select this if you are building a kernel for a desktop or embedded
   system with latency requirements in the milliseconds range.

We have found that on our test server (16-core server Dell 2950 with 6
RAID6 shelves) with the SL kernel, under load, the "ksoftirqd" daemon
takes up 100% CPU (this is a kernel thread involved in preemption),
and the xrootd process is only able to deliver data at about 500MB/sec
(about 50% of theoretical max on a on a 10Gb link).   

With the UL kernel, the "ksoftirq" CPU usage goes away completely, and
xrootd is easily able to deliver data at wire speed (~1000MB/sc)

	   - Charles