 |
 |
Hi Brian, That's weird, what happened to the 'id="sched"' part of the statistics? Andy -----Original Message----- From: Brian Bockelman Sent: Wednesday, July 06, 2011 4:17 PM To: Andrew Hanushevsky Cc: xrootd-dev Subject: Re: Hitting thread limits? On Jul 6, 2011, at 6:05 PM, Andrew Hanushevsky wrote: > Hi Brian, > > OK, so at the moment there really isn't anything you can do. The message > looks more ominous than need be. It just means that there is now an > internal queue of requests building up. So,things aren't as responsive as > they could be. The message gets repeated every 4K tries of getting a new > thread. Would be interesting to see how often the message goes out. > Just once: it set off the nagios alert as tests stopped working, and we took corrective action. > The longer term solution is to run more than one global redirector and set > then up in load balancing mode. > Noted. Will put on the TODO list. > An even longer term solution is to not run each select in a separate > thread but simply have a fixed pool of threads that execute that code. > This is the first time I've seen you over-run the redirector which means > you should be getting thousands of requests per second. Yeah, that sounds about right: from "lsof", there were 500 or so clients. Each client will make multiple queries at startup. > Can you send me two summary statistics each separated by about 10 seconds? > Use the xrd command to connect to the redirector xrootd and issue "query 1 > a". We fixed the underlying problem; I bet the statistics below are quite boring. Brian <statistics tod="1309993779" ver="v20110517-c443225" src="xrootd.unl.edu:1094" tos="1307994744" pgm="xrootd" ins="MAIN" pid="17133"><stats id="info"><host>xrootd.unl.edu</host><port>1094</port><name>MAIN</name></stats><stats id="buff"><reqs>177314</reqs><mem>1901568</mem><buffs>1685</buffs><adj>0</adj></stats><stats id="link"><num>460</num><maxn>0</maxn><tot>177318</tot><in>2968025505</in><out>3941314254</out><ctime>592626354</ctime><tmo>158307</tmo><stall>0</stall><sfps>1957955834</sfps></stats><stats id="poll"><att>459</att><en>20341649</en><ev>20341295</ev><int>0</int></stats><stats id="proc"><usr><s>246</s><u>95380</u></usr><sys><s>456</s><u>928556</u></sys></stats><stats id="xrootd"><num>177300</num><ops><open>20412434</open><rf>2</rf><rd>0</rd><pr>0</pr><wr>0</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>323027</misc></ops><aio><num>0</num><max>0</max><rej>0</rej></aio></stats><stats id="ofs"><role>meta manager</role><opr>0</opr><opw>0</opw><opp>0</opp><ups>0</ups><han>0</han><rdr>229863</rdr><bx root://xrootd.unl.edu:1094//>query 1 a <statistics tod="1309993784" ver="v20110517-c443225" src="xrootd.unl.edu:1094" tos="1307994744" pgm="xrootd" ins="MAIN" pid="17133"><stats id="info"><host>xrootd.unl.edu</host><port>1094</port><name>MAIN</name></stats><stats id="buff"><reqs>177315</reqs><mem>1901568</mem><buffs>1685</buffs><adj>0</adj></stats><stats id="link"><num>461</num><maxn>0</maxn><tot>177319</tot><in>2968025505</in><out>3941314254</out><ctime>592626354</ctime><tmo>158307</tmo><stall>0</stall><sfps>1957955834</sfps></stats><stats id="poll"><att>460</att><en>20341988</en><ev>20341634</ev><int>0</int></stats><stats id="proc"><usr><s>246</s><u>111381</u></usr><sys><s>456</s><u>936556</u></sys></stats><stats id="xrootd"><num>177301</num><ops><open>20412773</open><rf>2</rf><rd>0</rd><pr>0</pr><wr>0</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>323030</misc></ops><aio><num>0</num><max>0</max><rej>0</rej></aio></stats><stats id="ofs"><role>meta manager</role><opr>0</opr><opw>0</opw><opp>0</opp><ups>0</ups><han>0</han><rdr>229864</rdr><b