Hi Lukasz,
OK, there are several issues with serializing I/O through a single socket.
This has nothing to do with virtual streams as they are tied to the
realities of a physical socket. You can't design away those physical
limitations. Let's start with the obvious to the least obvious.
1) All bytes are transferred through a single pipe. That means if I
have two clients and one is reading 1K blocks and the other is reading 1M
blocks the 1K block client suffers the same latency as if it had been
reading 1M blocks. Same is true for writing. One xrdcp can destroy
performance for everybody.
2) Should a TCP packet get dropped and we go into packet recovery
then all clients see the increased latency. Since we have everything going
through a single pipe the odds of that happening increase.
3) Meta operations are, by their nature, serialized by the server.
That means they wpuld impact all clients whenever they occur. Since
some may take a bit of time to do there may be a substantial delay that
penalizes all clients that simply want to do I/O.
The extra overhead in doing authentication is no worse than the each
client contacting the server directly as would be the case without a
proxy. So, that argument is moot. In fact, most sites employ a simple
authentication scheme between a proxy and a server which makes the
overhead negligible compared to the downside of serializing all the
traffic through a single physical socket.
A physical single socket severely limits any kind of parallelism in most
cases. Virtual streams buy you nothing in this case. The bytes are bound
to a single physical communication channel that works serially by its
design. That means performance is bound by the slowest client.
In the end, request/response serialization is just a plain bad design.
Using multiple sockets with, in this case, a client-side idle timeout (I
wish I had thought of that side-effect on the proxy during the client
redesign) is the only performance enhancing solution here.
Andy
On Mon, 2 Sep 2013, Lukasz Janyst wrote:
> Hi Andy,
>
> could you please elaborate on the serialization issue? We have virtual
> streams and the new client fully supports them not only for data requests but
> for _all_ kinds of requests. So, I don't really see this serialization you're
> talking about. In fact, I would expect multiple streams to have worse
> performance due to constant re-authentication.
>
> Cheers,
> Lukasz
>
> On 01.09.2013 05:28, Andrew Hanushevsky wrote:
>> OK, this is what started the whole problem and a dive into the wrong
>> direction of serializing everything in the proxy server. Indeed, a long
>> time ago Lukasz and I discussed whether the client should close idle
>> connections and I said why bother, it would simplify the code by just
>> allowing the server to do it. While that was the right response for
>> standard client jobs it was wrong for long-lived servers that act like
>> clients (i.e. proxy server). First, the server-side timeout is set off
>> by default. You need to enable it and few people do as it rarely causes
>> a problem (sites with badly behaving VM hypervisors normally turn the
>> timeout on). Second, I didn't consider the side-effect on the proxy
>> server. So, what to do? I suspect the only real solution to this issue
>> is to implement an idle timeout in the new client. It's likely that we
>> would need two timeouts (just like in the old client) -- one for
>> redirectors (a longish timeout) and one for servers (a shortisj
>> timeout). The actual values would be controlled by some "envar".
>>
>> Lukasz, what's the possibility of adding this?
>>
>> Andy
>>
>> On Thu, 22 Aug 2013, Matevz Tadel wrote:
>>
>>> Hi Lukasz, Andy,
>>>
>>> On 8/22/13 7:52 AM, Lukasz Janyst wrote:
>>>> Hi Alja,
>>>>
>>>> that's a feature. The connection manager used to have idle time
>>>> after which
>>>> the connections were closed but Andy's preference was to manage
>>>> connection life
>>>> from the server-side.
>>>
>>> Hmmh ... then it seems the servers are not doing a too good job :( If
>>> you look at the netstat dump, there are 105 connections to CMS
>>> meta-manager at xrootd.unl.edu alone and 150 connections to various
>>> data servers ... and this is after replaying a one-day-worth of access
>>> from UCSD (320 files) and waiting for /three days/. So, the servers
>>> don't close all of them.
>>>
>>> The problem here is that proxy process stays up for a long time and it
>>> instantiates a new XrdCl for every file it is downloading.
>>>
>>> Andy, another problem here is with detailed monitoring ... I never get
>>> the last message of the 't' stream, the one containing the close so
>>> monitor thinks the files are still kept opened. As there is no
>>> disconnect and there are is no new activity on the channel, the
>>> buffers keeps sitting on the server. If this stays like this ... can I
>>> ask for 'close' event to cause flushing of the 't' stream when io[v]
>>> is in effect?
>>>
>>> Cheers,
>>> Matevz
>>>
>>>
>>>> Cheers,
>>>> Lukasz
>>>>
>>>> On 22.08.2013 16:15, Alja Mrak-Tadel wrote:
>>>>> Hello,
>>>>>
>>>>> I'm testing a proxy server with new client (branch xrdposixcl) and I
>>>>> noticed that socket connections stay open for days. Here is an example
>>>>> of netstat command on the machine I run xrootd proxy:
>>>>> http://uaf-2.t2.ucsd.edu/~alja/netstat.txt
>>>>> Notice also there are several connections to the meta manager,
>>>>> xrootd.unl.edu.
>>>>>
>>>>> From xrootd log http://uaf-2.t2.ucsd.edu/~alja/proxy.log I can see
>>>>> XrdCl::File::Close() is called but it is difficult to trace why sockets
>>>>> are left open.
>>>>>
>>>>> I'm working on a caching proxy server, where it is important that
>>>>> connections get closed reasonably soon ... the proxy is potentially
>>>>> accessing all Xrootd servers in a federation.
>>>>>
>>>>> Thanks,
>>>>> Alja
>>>>>
>>>>> ########################################################################
>>>>>
>>>>> Use REPLY-ALL to reply to list
>>>>>
>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|