Dear xrootd experts,
we upgraded to version 5.4.2 of xrootd two days ago.
It worked flawlessly for 1.5 days, but now cmsd spawns thousands of threads. This causes huge loads (up to 30.000) on the machine.
Restarting the daemon does not solve the problem.
Extra info: we are part of the xrootd federation of the cms experiment.
Do you have any idea how we can fix/debug this problem? The logs do not tell us a lot. It's mostly requests for files in the cmsd logs [1] and nothing special in the xrootd logs [2]
Any help is greatly appreciated.
Olivier.
[1] cmsd.log
220519 15:42:40 22203 cms_Dispatch: manager.0:[log in to unmask] for state dlen=156
220519 15:42:40 6266 manager.0:[log in to unmask] cms_do_State: /store/user/nshadski/c
ustomNano/QCD_Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016p
ostVFP_NanoAODv9_1-32.root
220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] for state dlen=156
220519 15:42:40 6267 manager.0:[log in to unmask] cms_do_State: /store/user/nshadski/customNano/QCD
_Pt_470to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAO
Dv9_1-32.root
220519 15:42:40 22265 cms_Dispatch: manager.0:[log in to unmask] for state dlen=105
220519 15:42:40 6268 manager.0:[log in to unmask] cms_do_State: /store/data/Run2018D/EGamma/MINIAOD
/12Nov2019_UL2018-v4/260000/E8C2D279-4422-E743-904A-4233F0BF230E.root
220519 15:42:40 22266 cms_Dispatch: manager.0:[log in to unmask] for state dlen=156
220519 15:42:40 6269 manager.0:[log in to unmask] cms_do_State: /store/user/nshadski/customNano/QCD_Pt_47
0to600_TuneCP5_13TeV_pythia8/KIT_CustomNanoV9_MC_2016postVFP/211229_123611/0000/MC_2016postVFP_NanoAODv9_1-
32.root
[2] xrootd.log
220519 15:42:30 22178 sysThrottleManager: Current IO counter is 0; total IO wait time is 0ms.
220519 15:42:31 22178 sysThrottleManager: Round ops allocation -1