 |
 |
Hi Andy, Thanks for bearing with us through all this thread :) On 08/31/13 20:28, Andrew Hanushevsky wrote: > OK, this is what started the whole problem and a dive into the wrong direction > of serializing everything in the proxy server. Indeed, a long time ago Lukasz > and I discussed whether the client should close idle connections and I said why > bother, it would simplify the code by just allowing the server to do it. While > that was the right response for standard client jobs it was wrong for long-lived > servers that act like clients (i.e. proxy server). First, the server-side > timeout is set off by default. You need to enable it and few people do as it > rarely causes a problem (sites with badly behaving VM hypervisors normally turn > the timeout on). Second, I didn't consider the side-effect on the proxy server. > So, what to do? I suspect the only real solution to this issue is to implement > an idle timeout in the new client. It's likely that we would need two timeouts > (just like in the old client) -- one for redirectors (a longish timeout) and one > for servers (a shortisj timeout). The actual values would be controlled by some > "envar". Keep in mind that if we force creation of sockets for each client, it might not make much sense to keep them alive ... or old, non-used sockets can get reused by new clients? Will this be properly reported in the monitoring 'u' stream? Andy, within the discussion I also asked for 't' stream to be flushed on close event. This way monitoring can be done with the file even though the session still remains opened for unknown period of time. > Lukasz, what's the possibility of adding this? I'd also like to ask for this special case ... if timeouts are set to 0, the all sockets used by a client should be closed on the spot. In the old client there were weird interferences that practically made sockets live for hours, sometimes days, even if the timeout was set to 10 minutes. Matevz > Andy > > On Thu, 22 Aug 2013, Matevz Tadel wrote: > >> Hi Lukasz, Andy, >> >> On 8/22/13 7:52 AM, Lukasz Janyst wrote: >>> Hi Alja, >>> >>> that's a feature. The connection manager used to have idle time after which >>> the connections were closed but Andy's preference was to manage connection life >>> from the server-side. >> >> Hmmh ... then it seems the servers are not doing a too good job :( If you look >> at the netstat dump, there are 105 connections to CMS meta-manager at >> xrootd.unl.edu alone and 150 connections to various data servers ... and this >> is after replaying a one-day-worth of access from UCSD (320 files) and waiting >> for /three days/. So, the servers don't close all of them. >> >> The problem here is that proxy process stays up for a long time and it >> instantiates a new XrdCl for every file it is downloading. >> >> Andy, another problem here is with detailed monitoring ... I never get the >> last message of the 't' stream, the one containing the close so monitor thinks >> the files are still kept opened. As there is no disconnect and there are is no >> new activity on the channel, the buffers keeps sitting on the server. If this >> stays like this ... can I ask for 'close' event to cause flushing of the 't' >> stream when io[v] is in effect? >> >> Cheers, >> Matevz >> >> >>> Cheers, >>> Lukasz >>> >>> On 22.08.2013 16:15, Alja Mrak-Tadel wrote: >>>> Hello, >>>> >>>> I'm testing a proxy server with new client (branch xrdposixcl) and I >>>> noticed that socket connections stay open for days. Here is an example >>>> of netstat command on the machine I run xrootd proxy: >>>> http://uaf-2.t2.ucsd.edu/~alja/netstat.txt >>>> Notice also there are several connections to the meta manager, >>>> xrootd.unl.edu. >>>> >>>> From xrootd log http://uaf-2.t2.ucsd.edu/~alja/proxy.log I can see >>>> XrdCl::File::Close() is called but it is difficult to trace why sockets >>>> are left open. >>>> >>>> I'm working on a caching proxy server, where it is important that >>>> connections get closed reasonably soon ... the proxy is potentially >>>> accessing all Xrootd servers in a federation. >>>> >>>> Thanks, >>>> Alja >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-DEV list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-DEV list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1