 |
 |
Hi Olivier, This is to be expected when you join a federation and all of a sudden have to deal with 10 of thousands new requests. There are several mitigations, each with it's own set of drawbacks. I would suggest you consult with cms experts on some of these. a) You can artificially restrict the number of threads in the cmsd. The drawback here is that there is no distunction between local requests and federated requests and you may wind up crowding out local requests. Use the xrd.sched directive for this: https://xrootd.slac.stanford.edu/doc/dev55/xrd_config.htm#_Toc88513978 b) You can specify the percentage of federated requests you are wiling to handle. The is not as precise as it seems as the global redirector might not have any choice but to violate your request if your site has is the only source of a file or when every other site has reached its global share. Use the cms.sched directive for this: https://xrootd.slac.stanford.edu/doc/dev55/xrd_config.htm#_Toc88513978 c) Scale up to meet the demand. Nothing stops you from running a cluster of redirectors for the federation. By using the all.manager all option the load is equally split amongst all of the available sub-redirectors. The drawback is that you need more hardware to do this but given the load you are experiencing that's the only solution available without constaining the local resource. See the all.manager directive: https://xrootd.slac.stanford.edu/doc/dev54/cms_config.htm#_Toc53611061 The issue here is that your current setup is insuffient for the number of requests the federation is trying to handle. I would suggest talking to the cms federation managers to see if you can employ one of the above options or whether they have other alternatives. Andy On Thu, 19 May 2022, Olivier Devroede wrote: > Dear xrootd experts, > > we upgraded to version 5.4.2 of xrootd two days ago. > > It worked flawlessly for 1.5 days, but now cmsd spawns thousands of > threads. This causes huge loads (up to 30.000) on the machine. > Restarting the daemon does not solve the problem. > > Extra info: we are part of the xrootd federation of the cms experiment. > > Do you have any idea how we can fix/debug this problem? The logs do not > tell us a lot. It's mostly requests for files in the cmsd logs [1] and > nothing special in the xrootd logs [2] > > Any help is greatly appreciated. > > Olivier. > > [1] cmsd.log > > 220519 15:42:40 22203 cms_Dispatch: > manager.0:[log in to unmask] for state dlen=156 > 220519 15:42:40 6266 manager.0:[log in to unmask] > cms_do_State: /store/user/nshadski/c > ustomNano/QCD_Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016p > ostVFP_NanoAODv9_1-32.root > 220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] > for state dlen=156 > 220519 15:42:40 6267 manager.0:[log in to unmask] cms_do_State: > /store/user/nshadski/customNano/QCD > _Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAO > Dv9_1-32.root > 220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] > for state dlen=105 > 220519 15:42:40 6268 manager.0:[log in to unmask] cms_do_State: > /store/data/Run2018D/EGamma/MINIAOD > /12Nov2019_UL2018-v4/260000/E8C2D279-4422-E743-904A-4233F0BF230E.root > 220519 15:42:40 22266 cms_Dispatch: manager.0:[log in to unmask] for > state dlen=156 > 220519 15:42:40 6269 manager.0:[log in to unmask] cms_do_State: > /store/user/nshadski/customNano/QCD_Pt_47 > 0to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAODv9_1- > 32.root > > [2] xrootd.log > 220519 15:42:30 22178 sysThrottleManager: Current IO counter is 0; total IO > wait time is 0ms. > 220519 15:42:31 22178 sysThrottleManager: Round ops allocation -1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1