If it is still relevant ... Here is a link to xrootd log with XrdCl in
debug log level:
https://amraktad.web.cern.ch/amraktad/mail/xrdcl.log
Alja
On 8/26/13 7:37 PM, Matevz Tadel wrote:
> Great, thanks Lukasz!
>
> Matevz
>
> On 8/26/13 2:55 PM, Lukasz Janyst wrote:
>> I am actually also away. It looks like something in XrdPosix is
>> forcing new
>> connections. I will have a closer look once I am back on Wednesday.
>>
>> Lukasz
>>
>> On 8/26/13 11:46 PM, Lukasz Janyst wrote:
>>> I call it a trick because the only context I have seen it used in is
>>> forcing
>>> new physical connections instead of reusing existing ones.
>>>
>>> Lukasz
>>>
>>> On 8/26/13 11:45 PM, Lukasz Janyst wrote:
>>>> Hi Matevz,
>>>>
>>>> Andy's on vacation until 2nd Sep.
>>>>
>>>> Lukasz
>>>>
>>>> On 8/26/13 11:28 PM, Matevz Tadel wrote:
>>>>> Hi Lukasz, Everybody,
>>>>>
>>>>> On 8/24/13 5:55 AM, Lukasz Janyst wrote:
>>>>>> Yes, in fact the username "trick" to trigger the new connections
>>>>>> would still
>>>>>> work with the new client and that's likely what is happening here.
>>>>>
>>>>> Hmmh ... why do you call it a trick? And where are the usernames
>>>>> coming
>>>>> from? This is all running within a single xrootd-proxy process ...
>>>>> so it is
>>>>> always the same user, I don't see how this could change in any
>>>>> way. (On
>>>>> server side and in monitoring, this is obviously a different user,
>>>>> as port
>>>>> number is also combined into it.)
>>>>>
>>>>> Anyway ... this seems a really bad idea for the proxy :)
>>>>>
>>>>> Here is a dump of server/monitoring-side usernames for a bunch of
>>>>> consecutive file-close events. You can see some of them get reused
>>>>> ... but
>>>>> 105 of such connections stayed for 3 days+ (after reading ~300
>>>>> files with
>>>>> 300 XrdCls in a single process).
>>>>>
>>>>> * 413974 * alja.2636:[log in to unmask] *
>>>>> * 415017 * alja.2636:[log in to unmask] *
>>>>> * 416011 * alja.2636:[log in to unmask] *
>>>>> * 416476 * alja.2636:[log in to unmask] *
>>>>> * 417680 * alja.2636:[log in to unmask] *
>>>>> * 419305 * alja.2636:[log in to unmask] *
>>>>> * 419376 * alja.2636:[log in to unmask] *
>>>>> * 419520 * alja.2636:[log in to unmask] *
>>>>> * 419599 * alja.2636:[log in to unmask] *
>>>>> * 419942 * alja.2636:[log in to unmask] *
>>>>> * 420628 * alja.2636:[log in to unmask] *
>>>>> * 421065 * alja.2636:[log in to unmask] *
>>>>> * 422202 * alja.2636:[log in to unmask] *
>>>>> * 422688 * alja.2636:[log in to unmask] *
>>>>> * 423059 * alja.2636:[log in to unmask] *
>>>>> * 423080 * alja.2636:[log in to unmask] *
>>>>> * 424121 * alja.2636:[log in to unmask] *
>>>>> * 424264 * alja.2636:[log in to unmask] *
>>>>> * 431576 * alja.2636:[log in to unmask] *
>>>>> * 488603 * alja.2636:[log in to unmask] *
>>>>> * 489027 * alja.2636:[log in to unmask] *
>>>>> * 489340 * alja.2636:[log in to unmask] *
>>>>> * 494316 * alja.2636:[log in to unmask] *
>>>>> * 494699 * alja.2636:[log in to unmask] *
>>>>> * 496100 * alja.2636:[log in to unmask] *
>>>>>
>>>>>> In fact you can see what is exactly happening if you set the
>>>>>> XRD_LOGLEVEL and
>>>>>> XRD_LOGFILE envvars as described in the xrdcopy man page.
>>>>>
>>>>> Alja will run our test with this on ... but whatever the outcome,
>>>>> these
>>>>> connections should still be closed at some point.
>>>>>
>>>>> Andy, can you please comment on this? [Does anybody know, is Andy
>>>>> away ...
>>>>> or I should kick his chair in a separate email?]
>>>>>
>>>>> Cheers,
>>>>> Matevz
>>>>>
>>>>>> Cheers,
>>>>>> Lukasz
>>>>>>
>>>>>> On 23.08.2013 21:58, Fabrizio Furano wrote:
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> you are reminding to me a historical thing with this thread.
>>>>>>> You may
>>>>>>> want to doublecheck these two points, and see if the defaults or
>>>>>>> the
>>>>>>> behavior is appropriate to the behavior of your proxy:
>>>>>>>
>>>>>>> - by choice the connections to the redirectors had a very long
>>>>>>> TTL,
>>>>>>> one day if I remember correctly
>>>>>>> - at some point, due to some interaction with the sec (don't
>>>>>>> remember
>>>>>>> exactly what by now) the behavior was changed to having one
>>>>>>> physical
>>>>>>> connection per process per user. Maybe Gerri remembers the
>>>>>>> rationale of
>>>>>>> this better than me. My point is to raise that if one sees many
>>>>>>> phyconns
>>>>>>> from the same process, they could be linked to different user
>>>>>>> ids. Worth
>>>>>>> checking IMHO.
>>>>>>>
>>>>>>> Hope that helps.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Fabrizio
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 8/23/13 9:42 PM, Brian Bockelman wrote:
>>>>>>>> On Aug 23, 2013, at 2:05 PM, Alja Mrak Tadel
>>>>>>>> <[log in to unmask]> wrote:
>>>>>>>>
>>>>>>>>>>> normally XrdCl would open one connection per server it
>>>>>>>>>>> contacts.
>>>>>>>>>>> Not per file.
>>>>>>>>>>
>>>>>>>>>> OK, how do we get 105 connections from a single process to the
>>>>>>>>>> meta-manager then? :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> FYI:: The connections listed with netstat are to the origin
>>>>>>>>> xrootd.unl.edu.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> So - we see duplicate connections to the redirector, but no
>>>>>>>> duplicate
>>>>>>>> connections to the data server?
>>>>>>>>
>>>>>>>> Could be a client bug, of course!
>>>>>>>>
>>>>>>>> Brian
>>>>>>>> ########################################################################
>>>>>>>>
>>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>>
>>>>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|