Yes, that what we do for the BaBar farm at SLAC. It run's VM's and the way 
they stop a stop is by issuing a kell -9 on the VM. The hypervisor doesn't 
cleanup the connection in that case. The timeout then closes idle 
connections. The suggested value is a bit too large. I think we use 
something like a day as most jobs don't sit on an idle connection for that 
long. That said, I think Brian just doesn't want to deal with another 
config change.

Andy

On Tue, 2 Sep 2014, Andreas-Joachim Peters wrote:

> Why don't you just set a very conservative idle timeout on the server by
> default. This does not really harm and cleans stale connections of VMs,
> right?
>
> xrd.timeout idle 604800
>
> Cheers Andreas.
>
>
>
>
>
> On Tue, Sep 2, 2014 at 3:59 PM, Brian Bockelman <[log in to unmask]>
> wrote:
>
>> On Sep 2, 2014, at 2:34 AM, Andrew Hanushevsky <[log in to unmask]>
>> wrote:
>>
>>> I guess I don't get what keepalive's would solve relative to the client
>> other than somewhat faster recovery in the rare case that a server goes
>> away. A lot of work for handling a <10% problem. The bigger problem is
>> client's going away and the server not being told that this has happened.
>> This is particularly bad when the client is a virtual machine as some
>> hyervisors handle this correctly and some do not. Firewalls and NAT boxes
>> make this even more problematic.
>>>
>>> I see the point of enabling keepalive by default. However,as a practical
>> measure, this actually is a big change as the one would need to implement a
>> way to turn it off (the current implementation simply allows you to turn it
>> on); let alone allowing a keepalive time specification.
>>>
>>> Additionally, I am not at all convinced that, at scale, it would
>> actually solve the problem. Brian are you always running with keepalive on
>> and it actually solves all of your vaporozing client issues?
>>
>> Well, saying it solves "all" is a big claim (and HTCondor doesn't provide
>> enough statistics for me to back up the claim anyway).  It does, however,
>> mitigate this to the point where we haven't had to spend time on the issue
>> for several months (since we deployed the relevant version).  When the
>> problem was originally fixed, we did collect enough statistics to say this
>> "solved" things at problem sites.
>>
>> *Note* that this doesn't solve the problem of an overloaded site network -
>> it just helps the server to not have to track broken connections.  If the
>> network device is overloaded, detecting and re-establishing a TCP
>> connection will not help.
>>
>> I agree the client-side change is mostly just allowing a quicker
>> recovery.  However, I think the server-side change is worth the hassle to
>> clear up dead connections.
>>
>> Since dead connections only cause problems in aggregate (i.e., we don't
>> need to tune keepalive down to 1 minute), why not:
>>
>> a) Always turn keepalive on; remove this as an option, and
>> b) Provide no mechanism to provide a keepalive time specification.
>>
>> Seems simpler and I can't think of any large downsides (although maybe
>> that's because I've only had 1 cup of coffee today).
>>
>> Brian
>>
>>>
>>> Andy
>>>
>>> P.S. I agrre that the keepalive mechanism in TCPwon't cause a
>> scalability issue, This is a particular issue with proxies and NAT boxes
>> that can't track all of the connections in real time. In this case you may
>> get a false indication that the client is dead. As I said, in the xroot
>> world that shouldn't matter as the client would simply reconnect.
>>>
>>> On Thu, 28 Aug 2014, Brian Bockelman wrote:
>>>
>>>> If you're going to enable keepalive in the client -
>>>>
>>>> You might want to think about manually tuning the keepalive timeouts
>> down from the defaults (2 hours).  I recently adjusted it down to around 5
>> minutes in HTCondor because 2 hours was "too late" to detect the disconnect
>> to recover the jobs.
>>>>
>>>> There's a socket option to do this in Linux (which travels under a
>> different name in Mac OS X... not sure about Solaris).  Again, we've not
>> seen any kernel scalability issues from doing this.
>>>>
>>>> Brian
>>>>
>>>> On Aug 28, 2014, at 5:09 AM, Lukasz Janyst <[log in to unmask]> wrote:
>>>>
>>>>> Hi Brian,
>>>>>
>>>>>   for the server-side, it is Andy's call.
>>>>>
>>>>>   We have seen silent disconnection problems with ALICE sites in the
>> past, this is why I set up the keepalive functionality for sockets in the
>> old client. I will do the same for the new one as well.
>>>>>
>>>>> Chers,
>>>>>  Lukasz
>>>>>
>>>>> On 08/25/2014 02:52 PM, Brian Bockelman wrote:
>>>>>> Hi Lukasz, all,
>>>>>>
>>>>>> Can we enable keepalive by default?  I don't look forward to the task
>> of asking every site for a configuration change.
>>>>>>
>>>>>> At least on the linux platform, we have observed the kernel is able
>> to handle tens-of-thousands of sockets with keepalive enabled; it doesn't
>> appear to be a scalability issue.  There doesn't appear to be any protocol
>> built-in features we could use on the server side (although this doesn't
>> appear to be needed on the client side).
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>> On Aug 25, 2014, at 2:08 AM, Lukasz Janyst <[log in to unmask]> wrote:
>>>>>>
>>>>>>> On 08/22/2014 06:59 PM, Matevz Tadel wrote:
>>>>>>>>> Does the Xrootd server at least enable TCP keepalive?  That'll
>> close
>>>>>>>>> out dead connections after 2 hours.
>>>>>>>>
>>>>>>>> I don't think so ... I see things hanging up to 24 hours easily
>> (when
>>>>>>>> collector decides to give up on the session). Can this timeout be
>> set at
>>>>>>>> socket creation time?
>>>>>>>
>>>>>>>  Typically, this is handled by the TCP stack, but the
>> routers/firewalls on the way often mess things up. To enable the OS
>> keepalive for xrootd sockets you need to ask for it:
>> http://xrootd.org/doc/prod/xrd_config.htm#_Toc310725344
>>>>>>>
>>>>>>> Cheers,
>>>>>>>  Lukasz
>>>>>>>
>>>>>>>
>> ########################################################################
>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>
>>>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>>>
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1