Hi Kyle,
The summary counts are consistent. A read request is 24 bytes in length.
So, we find
37184778462/1548943168 = 24.007 (as expected)
The average payload in response:
26610785518162/1548943168 = 17179 (which looks pretty good)
However, this is misleading. Looking at the sample of recvfrom's we find:
File# Offset Length
0600 0bc5 00000000 0000000000d74e00 0008a400 = 566272 bytes
1f00 0bc5 00000000 0000000000dff200 00003e00 = 15872 bytes
0500 0bc5 00000000 0000000000e03000 00004400 = 17408 bytes
0500 0bc5 00000000 0000000000e07400 00001400 = 5120 bytes
1f00 0bc5 00000000 0000000000e08800 00001400 = 5120 bytes
0600 0bc5 00000000 0000000000e09c00 00001400 = 5120 bytes
So, you likely do a lot of small reads with the very large reads pushing
up the average. That said, I would still suspect the monitoring at this
point.
Andy
On Fri, 24 Jun 2011, Kyle Fransham wrote:
> Hi Andy,
>
> On 06/23/2011 04:28 PM, Andrew Hanushevsky wrote:
>> Hi Kyle,
>>
>> What we need to do is find out what the inbound traffic really is. So, some
>> more questions:
>>
>> 1) On your graph is that bytes in/out or packets in/out. If this is packets
>> then the graph is likely correct and you are simply doing a lot of very
>> small reads.
> That's definitely bytes, not packets on the graph.
>> 2) To find out a bit more statistics you can connect to the xrootd server
>> using the xrd command. Do the following:
>>
>> xrd<the_xrootd_server>
>> query 1 lp
>> exit
>>
>> The find the values between<in></in> and<out></out> that will give you
>> number of bytes in and number out. We need to see if that is reasonable
>> compared to actual requests which you will find between<rd></rd> and
>> <wr></wr> (read/write counters).
>
> <rd>1548943168</rd>
> <wr>0</wr>
> <in>37184778462</in>
> <out>26610785518162</out>
>
> So writes are 0, as expected. Inbound traffic is three orders of magnitude
> less than outbound, which doesn't correspond to our monitoring, but looks
> good.
>
>> 3) If those still seem not to correspond then we can look at the actual
>> xrootd kernel calls using strace. For instance:
>>
>> strace -f -xx -ttt -p<pid> -e trace=network 2>&1 | grep 'recv('>
>> <outfile>
> Okay, so I see no calls to 'recv(' in the strace. However, I do see calls to
> 'recvfrom(' that look like this:
>
> [pid 16211] 1308923304.005638 recvfrom(19,
>
"\x06\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xd7\x4e\x00\x00\x08\xa4\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
> [pid 16211] 1308923304.045490 recvfrom(19,
> "\x1f\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xdf\xf2\x00\x00\x00\x3e\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
> [pid 16211] 1308923304.046110 recvfrom(19,
> "\x05\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x30\x00\x00\x00\x44\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
> [pid 16211] 1308923304.999048 recvfrom(19,
> "\x05\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x74\x00\x00\x00\x14\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
> [pid 16211] 1308923304.999260 recvfrom(19,
> "\x1f\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x88\x00\x00\x00\x14\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
> [pid 16211] 1308923305.000027 recvfrom(19,
> "\x06\x00\x0b\xc5\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe0\x9c\x00\x00\x00\x14\x00\x00\x00\x00\x00",
> 24, 0, NULL, NULL) = 24
>
>> This will capture server recv() requests (inbound traffic). No need to run
>> this more than a minute or two.
>>
>> 4) If that doesn't reveal anything then the only other option is that there
>> really is something else on that machine that is accepting incoming
>> traffic.
>> If it isn't udp then netstat should show you who that might be.
> At a first glance, netstat doesn't show any real flags. Also, xrootd really
> is the only thing running on this machine, and we see lots of input whenever
> a user runs jobs, (i.e. reads data from xrootd) and no input otherwise. So
> the two are at least correlated...
>
> Thanks for your help!
>
> Kyle
>> Andy
>>
>>
>> -----Original Message-----
>> From: Kyle Fransham
>> Sent: Thursday, June 23, 2011 1:05 PM
>> To: Andrew Hanushevsky
>> Cc: xrootd-l
>> Subject: Re: inbound traffic
>>
>> Hi Andy,
>>
>> This is a machine at UVic that we use to serve BaBar xrootd files to
>> virtual machines that we spawn in the cloud. It's running little else
>> besides xrootd. Any traffic on the external interface (the plot that I
>> sent) is xrootd. We see very high inbound traffic almost all of the time.
>>
>> On the back end, we have 10TB or so of data in a lustre filesystem
>> that's distributed across multiple workers. Since this is a distributed
>> filesystem, we expect (and we do see) traffic on the internal interface
>> that's associated with the reading of xrootd collections. But we don't
>> expect to see that externally...
>>
>> What else can I tell you about this machine/setup to help diagnose the
>> problem?
>>
>> Thanks,
>>
>> Kyle
>>
>> On 06/23/2011 03:36 PM, Andrew Hanushevsky wrote:
>>> Hi Kyle,
>>>
>>> There should be little inbound traffic unless that machine is used for
>>> more
>>> than just xrootd services. What machine are we talking about?
>>>
>>> Andy
>>>
>>> -----Original Message-----
>>> From: Kyle Fransham
>>> Sent: Thursday, June 23, 2011 7:48 AM
>>> To: xrootd-l
>>> Subject: inbound traffic
>>>
>>> Hi all,
>>>
>>> We've got a single xrootd server serving out BaBar root files over the
>>> WAN. We notice that there is a lot of inbound traffic, even though our
>>> files are exported read-only. Attached is a network plot showing the
>>> traffic on the xrootd interface for four simultaneous user analysis
>>> jobs. (In case you can't see the attachment, the inboud traffic tends
>>> to be about 75% of the outbound traffic.)
>>>
>>> Is this expected behaviour?
>>>
>>> Thanks,
>>>
>>> Kyle
>>>
>
