 |
 |
Hi Lukasz, thanks for feedback, yep, it looks like more the system-wide scalability issue which might or might not be connected to any bug in xrootd. Though, the xrootd is the service hammered by something here which needs attention, too. We'll watch closely SYNs on the UNL host and try to debug live when this occurs again. Thanks, Marian On 3/16/16 5:34 AM, Lukasz Janyst wrote: > One way to debug this would be to run wireshark to see where the bogus > SYN packets are coming from. > > Lukasz > > On Wed, Mar 16, 2016, at 11:27, Lukasz Janyst wrote: >> Isn't it a sign of either a DOS attack or a network problem? I would >> guess that a restart of the service helps because, by closing the >> listening socket, you close the corresponding kernel SYN queue. >> >> Lukasz >> >> On Wed, Mar 16, 2016, at 00:39, Marian Zvada wrote: >>> Hi Folks, >>> >>> we're seeing these two types of kernel messages which are obviously >>> connected to xrootd process on US regional redirectors running on the >>> port 1094: >>> >>> --- >>> kernel: TCPv6: Possible SYN flooding on port 1094. Sending cookies. >>> kernel: possible SYN flooding on port 1094. Sending cookies. >>> --- >>> >>> This is happening intermittently on both US regional redirectors >>> cmsxrootd1.fnal.gov and xrootd.unl.edu. Both are behind DNS aliased host >>> cmsxrootd.fnal.gov. We're pretty confident that this typically occurs in >>> syslog when redirector is giving very long waits for access to files >>> through xrootd. >>> >>> Simple restart of service bring response time back to normal. We also >>> didn't notice any significant increase in use of memory nor cpu on the >>> machines itself so we're wondering if anyone from the list or developers >>> may explain if this is something to worry about. It is also hard to >>> catch so maybe if you have any idea what to watch next time and record >>> (besides core file) that'll help. Luckily, we at least know when we're >>> getting warning state of the xrootd-fallback SAM test this 'flooding' is >>> likely happening again... >>> >>> FNAL and UNL regional redirectors run xrootd-4.3.0-0.rc3.el6.x86_64 and >>> along slowness seen and odd kernel records in system logs there is >>> nothing obvious in the xrootd and cmsd logs to report. Maybe do you know >>> which specific xrootd process chain might trigger this kernel errors? >>> >>> Any feedback is very welcome! >>> >>> Thanks, >>> Marian >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1