Print

Print


Hi Marian,

the only other not-completely-improbable explanation is that a massive
number of clients tries to connect within a time slice of one RTT.

Cheers,
   Lukasz

On Wed, Mar 16, 2016, at 20:58, Marian Zvada wrote:
> Hi Lukasz,
> 
> thanks for feedback, yep, it looks like more the system-wide scalability 
> issue which might or might not be connected to any bug in xrootd. 
> Though, the xrootd is the service hammered by something here which needs 
> attention, too.
> 
> We'll watch closely SYNs on the UNL host and try to debug live when this 
> occurs again.
> 
> Thanks,
> Marian
> 
> On 3/16/16 5:34 AM, Lukasz Janyst wrote:
> > One way to debug this would be to run wireshark to see where the bogus
> > SYN packets are coming from.
> >
> >     Lukasz
> >
> > On Wed, Mar 16, 2016, at 11:27, Lukasz Janyst wrote:
> >> Isn't it a sign of either a DOS attack or a network problem? I would
> >> guess that a restart of the service helps because, by closing the
> >> listening socket, you close the corresponding kernel SYN queue.
> >>
> >>     Lukasz
> >>
> >> On Wed, Mar 16, 2016, at 00:39, Marian Zvada wrote:
> >>> Hi Folks,
> >>>
> >>> we're seeing these two types of kernel messages which are obviously
> >>> connected to xrootd process on US regional redirectors running on the
> >>> port 1094:
> >>>
> >>> ---
> >>> kernel: TCPv6: Possible SYN flooding on port 1094. Sending cookies.
> >>> kernel: possible SYN flooding on port 1094. Sending cookies.
> >>> ---
> >>>
> >>> This is happening intermittently on both US regional redirectors
> >>> cmsxrootd1.fnal.gov and xrootd.unl.edu. Both are behind DNS aliased host
> >>> cmsxrootd.fnal.gov. We're pretty confident that this typically occurs in
> >>> syslog when redirector is giving very long waits for access to files
> >>> through xrootd.
> >>>
> >>> Simple restart of service bring response time back to normal. We also
> >>> didn't notice any significant increase in use of memory nor cpu on the
> >>> machines itself so we're wondering if anyone from the list or developers
> >>> may explain if this is something to worry about. It is also hard to
> >>> catch so maybe if you have any idea what to watch next time and record
> >>> (besides core file) that'll help. Luckily, we at least know when we're
> >>> getting warning state of the xrootd-fallback SAM test this 'flooding' is
> >>> likely happening again...
> >>>
> >>> FNAL and UNL regional redirectors run xrootd-4.3.0-0.rc3.el6.x86_64 and
> >>> along slowness seen and odd kernel records in system logs there is
> >>> nothing obvious in the xrootd and cmsd logs to report. Maybe do you know
> >>> which specific xrootd process chain might trigger this kernel errors?
> >>>
> >>> Any feedback is very welcome!
> >>>
> >>> Thanks,
> >>> Marian
> >>>
> >>> ########################################################################
> >>> Use REPLY-ALL to reply to list
> >>>
> >>> To unsubscribe from the XROOTD-L list, click the following link:
> >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1