 |
 |
Hi Olivier, It sure is a conincidence. The cmsd code base has not materially changed since 5.4.0. What release did you upgrade from? Andy On Fri, 20 May 2022, Olivier Devroede wrote: > Hi Andrew, > > Thanks a lot for your answer. > > I finally had to go far away from the default and have set this: > xrd.sched mint 8 maxt 102 avlt 25 idle 39 > > And now, it is stable, and the ETF tests run. I'll experiment with the > cms.sched later, but I want to have some stability for the time being. And > see if everyone gets served. > > That being said, we are not new to the federation. We have been part of it > for many years and only occasionally had the cmsd go crazy like this. And > in the past, waiting for a few hours always solved the problem. But not > this time. > It might be a coincidence, but this happened 1.5 days after upgrading to > xrootd 5.4.2. > > > Kind regards, > > Olivier. > > On Fri, 20 May 2022 at 01:55, Andrew Hanushevsky <[log in to unmask]> > wrote: > >> Hi Olivier, >> >> This is to be expected when you join a federation and all of a sudden have >> to deal with 10 of thousands new requests. There are several mitigations, >> each with it's own set of drawbacks. I would suggest you consult with cms >> experts on some of these. >> >> a) You can artificially restrict the number of threads in the cmsd. The >> drawback here is that there is no distunction between local requests and >> federated requests and you may wind up crowding out local requests. Use >> the xrd.sched directive for this: >> https://xrootd.slac.stanford.edu/doc/dev55/xrd_config.htm#_Toc88513978 >> >> b) You can specify the percentage of federated requests you are wiling to >> handle. The is not as precise as it seems as the global redirector might >> not have any choice but to violate your request if your site has is the >> only source of a file or when every other site has reached its global >> share. Use the cms.sched directive for this: >> https://xrootd.slac.stanford.edu/doc/dev55/xrd_config.htm#_Toc88513978 >> >> c) Scale up to meet the demand. Nothing stops you from running a cluster >> of redirectors for the federation. By using the all.manager all option >> the load is equally split amongst all of the available sub-redirectors. >> The drawback is that you need more hardware to do this but given the load >> you are experiencing that's the only solution available without >> constaining the local resource. See the all.manager directive: >> https://xrootd.slac.stanford.edu/doc/dev54/cms_config.htm#_Toc53611061 >> >> The issue here is that your current setup is insuffient for the number of >> requests the federation is trying to handle. I would suggest talking to >> the cms federation managers to see if you can employ one of the above >> options or whether they have other alternatives. >> >> Andy >> >> >> On Thu, 19 May 2022, Olivier Devroede wrote: >> >>> Dear xrootd experts, >>> >>> we upgraded to version 5.4.2 of xrootd two days ago. >>> >>> It worked flawlessly for 1.5 days, but now cmsd spawns thousands of >>> threads. This causes huge loads (up to 30.000) on the machine. >>> Restarting the daemon does not solve the problem. >>> >>> Extra info: we are part of the xrootd federation of the cms experiment. >>> >>> Do you have any idea how we can fix/debug this problem? The logs do not >>> tell us a lot. It's mostly requests for files in the cmsd logs [1] and >>> nothing special in the xrootd logs [2] >>> >>> Any help is greatly appreciated. >>> >>> Olivier. >>> >>> [1] cmsd.log >>> >>> 220519 15:42:40 22203 cms_Dispatch: >>> manager.0:[log in to unmask] for state dlen=156 >>> 220519 15:42:40 6266 manager.0:[log in to unmask] >>> cms_do_State: /store/user/nshadski/c >>> >> ustomNano/QCD_Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016p >>> ostVFP_NanoAODv9_1-32.root >>> 220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] >>> for state dlen=156 >>> 220519 15:42:40 6267 manager.0:[log in to unmask] cms_do_State: >>> /store/user/nshadski/customNano/QCD >>> >> _Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAO >>> Dv9_1-32.root >>> 220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] >>> for state dlen=105 >>> 220519 15:42:40 6268 manager.0:[log in to unmask] cms_do_State: >>> /store/data/Run2018D/EGamma/MINIAOD >>> /12Nov2019_UL2018-v4/260000/E8C2D279-4422-E743-904A-4233F0BF230E.root >>> 220519 15:42:40 22266 cms_Dispatch: manager.0:[log in to unmask] for >>> state dlen=156 >>> 220519 15:42:40 6269 manager.0:[log in to unmask] cms_do_State: >>> /store/user/nshadski/customNano/QCD_Pt_47 >>> >> 0to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAODv9_1- >>> 32.root >>> >>> [2] xrootd.log >>> 220519 15:42:30 22178 sysThrottleManager: Current IO counter is 0; total >> IO >>> wait time is 0ms. >>> 220519 15:42:31 22178 sysThrottleManager: Round ops allocation -1 >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1