Somebody should probably have a look at this. Forwarding for
documentation purposes.

   Lukasz

Begin forwarded message:

Date: Sun, 17 May 2015 23:19:47 +0200
From: Lukasz Janyst <[log in to unmask]>
To: Andreas-Joachim Peters <[log in to unmask]>
Subject: Re: 10G Question


Hi Andreas,

   this was not fixed, but from what I could tell this might actually
have been related to the particular version of kernel/driver.

   The situation is as follows:

1) If you only have one active socket on a 10G interface you get only
around 50% of bandwidth utilisation.
2) If you have two or more active sockets you saturate the link.

   Now, from my observation/research:

1) Socket watermarking (SO_RCVLOWAT sockopt and friends) is not
implemented/broken in Linux.
2) When a tiny bit of data arrives in the NIC buffer, the TCP stack
triggers a poll event for the descriptor. This results in a switch to
user space.
3) This tiny bit of data is then retrieved from the descriptor using a
read call on a non-blocking socket (another user-to-kernel-space switch
and back)
4) Another read request is issued (another trip to kernel space and
back) on the socket witch, this time, resutls with EWOULDBLOC errno
being set, and the client logic deciding to go back to the poll
syscall, which quickly gets triggered again and we're back at 2. The
cycle repeats itself and we're spending all of the time doing useless
context switches witch could be easily prevented if SO_RCVLOWAT
actually worked.

   For two or more sockets, the timing changes and the poll events are
triggered when larger chunks of data have been accumulated. The more
sockets/connections we have the better it gets.

   I vaguely recall that there was an ioctl option controlling this,
but I cannot find it now. Alternatively, you can have a look at the
driver or implement proper socket watermarking in the TCP stack of
Linux :)

   I hope this helps.

Cheers,
   Lukasz

On Wed, 13 May 2015 16:55:48 +0200
Andreas-Joachim Peters <[log in to unmask]> wrote:

> Hi Lukasz,
> 
> in case you remember something quickly.
> 
> We have some problems with the performance of copy commands over 10G.
> 
> If I compare the iperf performance with copy performance of xrootd I
> get only 50% with xrootd.
> 
> I played with the chunksize etc. but with the 3.3.6 versions it seems
> not to change much.
> 
> Do remember anything particular that had been fixed only with the 4.x
> version concerning window scaling etc. ??
> 
> I hope you are all well.
> 
> Cheers Andreas.

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1