 |
 |
OK, this is what started the whole problem and a dive into the wrong direction of serializing everything in the proxy server. Indeed, a long time ago Lukasz and I discussed whether the client should close idle connections and I said why bother, it would simplify the code by just allowing the server to do it. While that was the right response for standard client jobs it was wrong for long-lived servers that act like clients (i.e. proxy server). First, the server-side timeout is set off by default. You need to enable it and few people do as it rarely causes a problem (sites with badly behaving VM hypervisors normally turn the timeout on). Second, I didn't consider the side-effect on the proxy server. So, what to do? I suspect the only real solution to this issue is to implement an idle timeout in the new client. It's likely that we would need two timeouts (just like in the old client) -- one for redirectors (a longish timeout) and one for servers (a shortisj timeout). The actual values would be controlled by some "envar". Lukasz, what's the possibility of adding this? Andy On Thu, 22 Aug 2013, Matevz Tadel wrote: > Hi Lukasz, Andy, > > On 8/22/13 7:52 AM, Lukasz Janyst wrote: >> Hi Alja, >> >> that's a feature. The connection manager used to have idle time after >> which >> the connections were closed but Andy's preference was to manage connection >> life >> from the server-side. > > Hmmh ... then it seems the servers are not doing a too good job :( If you > look at the netstat dump, there are 105 connections to CMS meta-manager at > xrootd.unl.edu alone and 150 connections to various data servers ... and this > is after replaying a one-day-worth of access from UCSD (320 files) and > waiting for /three days/. So, the servers don't close all of them. > > The problem here is that proxy process stays up for a long time and it > instantiates a new XrdCl for every file it is downloading. > > Andy, another problem here is with detailed monitoring ... I never get the > last message of the 't' stream, the one containing the close so monitor > thinks the files are still kept opened. As there is no disconnect and there > are is no new activity on the channel, the buffers keeps sitting on the > server. If this stays like this ... can I ask for 'close' event to cause > flushing of the 't' stream when io[v] is in effect? > > Cheers, > Matevz > > >> Cheers, >> Lukasz >> >> On 22.08.2013 16:15, Alja Mrak-Tadel wrote: >>> Hello, >>> >>> I'm testing a proxy server with new client (branch xrdposixcl) and I >>> noticed that socket connections stay open for days. Here is an example >>> of netstat command on the machine I run xrootd proxy: >>> http://uaf-2.t2.ucsd.edu/~alja/netstat.txt >>> Notice also there are several connections to the meta manager, >>> xrootd.unl.edu. >>> >>> From xrootd log http://uaf-2.t2.ucsd.edu/~alja/proxy.log I can see >>> XrdCl::File::Close() is called but it is difficult to trace why sockets >>> are left open. >>> >>> I'm working on a caching proxy server, where it is important that >>> connections get closed reasonably soon ... the proxy is potentially >>> accessing all Xrootd servers in a federation. >>> >>> Thanks, >>> Alja >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-DEV list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-DEV list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1