 |
 |
If it is still relevant ... Here is a link to xrootd log with XrdCl in debug log level: https://amraktad.web.cern.ch/amraktad/mail/xrdcl.log Alja On 8/26/13 7:37 PM, Matevz Tadel wrote: > Great, thanks Lukasz! > > Matevz > > On 8/26/13 2:55 PM, Lukasz Janyst wrote: >> I am actually also away. It looks like something in XrdPosix is >> forcing new >> connections. I will have a closer look once I am back on Wednesday. >> >> Lukasz >> >> On 8/26/13 11:46 PM, Lukasz Janyst wrote: >>> I call it a trick because the only context I have seen it used in is >>> forcing >>> new physical connections instead of reusing existing ones. >>> >>> Lukasz >>> >>> On 8/26/13 11:45 PM, Lukasz Janyst wrote: >>>> Hi Matevz, >>>> >>>> Andy's on vacation until 2nd Sep. >>>> >>>> Lukasz >>>> >>>> On 8/26/13 11:28 PM, Matevz Tadel wrote: >>>>> Hi Lukasz, Everybody, >>>>> >>>>> On 8/24/13 5:55 AM, Lukasz Janyst wrote: >>>>>> Yes, in fact the username "trick" to trigger the new connections >>>>>> would still >>>>>> work with the new client and that's likely what is happening here. >>>>> >>>>> Hmmh ... why do you call it a trick? And where are the usernames >>>>> coming >>>>> from? This is all running within a single xrootd-proxy process ... >>>>> so it is >>>>> always the same user, I don't see how this could change in any >>>>> way. (On >>>>> server side and in monitoring, this is obviously a different user, >>>>> as port >>>>> number is also combined into it.) >>>>> >>>>> Anyway ... this seems a really bad idea for the proxy :) >>>>> >>>>> Here is a dump of server/monitoring-side usernames for a bunch of >>>>> consecutive file-close events. You can see some of them get reused >>>>> ... but >>>>> 105 of such connections stayed for 3 days+ (after reading ~300 >>>>> files with >>>>> 300 XrdCls in a single process). >>>>> >>>>> * 413974 * alja.2636:[log in to unmask] * >>>>> * 415017 * alja.2636:[log in to unmask] * >>>>> * 416011 * alja.2636:[log in to unmask] * >>>>> * 416476 * alja.2636:[log in to unmask] * >>>>> * 417680 * alja.2636:[log in to unmask] * >>>>> * 419305 * alja.2636:[log in to unmask] * >>>>> * 419376 * alja.2636:[log in to unmask] * >>>>> * 419520 * alja.2636:[log in to unmask] * >>>>> * 419599 * alja.2636:[log in to unmask] * >>>>> * 419942 * alja.2636:[log in to unmask] * >>>>> * 420628 * alja.2636:[log in to unmask] * >>>>> * 421065 * alja.2636:[log in to unmask] * >>>>> * 422202 * alja.2636:[log in to unmask] * >>>>> * 422688 * alja.2636:[log in to unmask] * >>>>> * 423059 * alja.2636:[log in to unmask] * >>>>> * 423080 * alja.2636:[log in to unmask] * >>>>> * 424121 * alja.2636:[log in to unmask] * >>>>> * 424264 * alja.2636:[log in to unmask] * >>>>> * 431576 * alja.2636:[log in to unmask] * >>>>> * 488603 * alja.2636:[log in to unmask] * >>>>> * 489027 * alja.2636:[log in to unmask] * >>>>> * 489340 * alja.2636:[log in to unmask] * >>>>> * 494316 * alja.2636:[log in to unmask] * >>>>> * 494699 * alja.2636:[log in to unmask] * >>>>> * 496100 * alja.2636:[log in to unmask] * >>>>> >>>>>> In fact you can see what is exactly happening if you set the >>>>>> XRD_LOGLEVEL and >>>>>> XRD_LOGFILE envvars as described in the xrdcopy man page. >>>>> >>>>> Alja will run our test with this on ... but whatever the outcome, >>>>> these >>>>> connections should still be closed at some point. >>>>> >>>>> Andy, can you please comment on this? [Does anybody know, is Andy >>>>> away ... >>>>> or I should kick his chair in a separate email?] >>>>> >>>>> Cheers, >>>>> Matevz >>>>> >>>>>> Cheers, >>>>>> Lukasz >>>>>> >>>>>> On 23.08.2013 21:58, Fabrizio Furano wrote: >>>>>>> Hi guys, >>>>>>> >>>>>>> you are reminding to me a historical thing with this thread. >>>>>>> You may >>>>>>> want to doublecheck these two points, and see if the defaults or >>>>>>> the >>>>>>> behavior is appropriate to the behavior of your proxy: >>>>>>> >>>>>>> - by choice the connections to the redirectors had a very long >>>>>>> TTL, >>>>>>> one day if I remember correctly >>>>>>> - at some point, due to some interaction with the sec (don't >>>>>>> remember >>>>>>> exactly what by now) the behavior was changed to having one >>>>>>> physical >>>>>>> connection per process per user. Maybe Gerri remembers the >>>>>>> rationale of >>>>>>> this better than me. My point is to raise that if one sees many >>>>>>> phyconns >>>>>>> from the same process, they could be linked to different user >>>>>>> ids. Worth >>>>>>> checking IMHO. >>>>>>> >>>>>>> Hope that helps. >>>>>>> >>>>>>> Cheers >>>>>>> Fabrizio >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 8/23/13 9:42 PM, Brian Bockelman wrote: >>>>>>>> On Aug 23, 2013, at 2:05 PM, Alja Mrak Tadel >>>>>>>> <[log in to unmask]> wrote: >>>>>>>> >>>>>>>>>>> normally XrdCl would open one connection per server it >>>>>>>>>>> contacts. >>>>>>>>>>> Not per file. >>>>>>>>>> >>>>>>>>>> OK, how do we get 105 connections from a single process to the >>>>>>>>>> meta-manager then? :) >>>>>>>>> >>>>>>>>> >>>>>>>>> FYI:: The connections listed with netstat are to the origin >>>>>>>>> xrootd.unl.edu. >>>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> So - we see duplicate connections to the redirector, but no >>>>>>>> duplicate >>>>>>>> connections to the data server? >>>>>>>> >>>>>>>> Could be a client bug, of course! >>>>>>>> >>>>>>>> Brian >>>>>>>> ######################################################################## >>>>>>>> >>>>>>>> Use REPLY-ALL to reply to list >>>>>>>> >>>>>>>> To unsubscribe from the XROOTD-DEV list, click the following link: >>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1