 |
 |
Hi JT, Could you submit a PR for that change? Andy On Fri, 14 Jan 2022, jthiltges wrote: > On the local redirector at Nebraska, we began seeing odd spikes in system load after the upgrade to v5.4.0. Load was quite high, reaching 1000-5000, with the system being mostly unresponsive. > > As Brian said, the problem appeared related to SSL negotiation overhead, where the server generates new DH parameters for each connection. > > <details> > <summary>Sample backtrace</summary> > > ``` > Thread 108 (Thread 0x7f4bbb1fe700 (LWP 28534)): > #0 BN_mod_word ***@***.***=0x7f4b0f911c00, ***@***.***=1699) at bn_word.c:102 > #1 0x00007f4bf8327431 in probable_prime_dh_safe (ctx=0x7f4b6bc2eac0, rem=0x7f4b2db70a98, padd=0x7f4b2db70a80, bits=511, p=0x7f4b0f911c00) at bn_prime.c:502 > #2 BN_generate_prime_ex (ret=0x7f4b0f911c00, ***@***.***=512, ***@***.***=1, ***@***.***=0x7f4b2db70a80, ***@***.***=0x7f4b2db70a98, ***@***.***=0x0) at bn_prime.c:186 > #3 0x00007f4bf83635e9 in dh_builtin_genparams (ret=0x7f4bdedd1ba0, ret=0x7f4bdedd1ba0, cb=0x0, generator=5, prime_len=512) at dh_gen.c:194 > #4 DH_generate_parameters_ex (ret=0x7f4bdedd1ba0, prime_len=512, generator=5, cb=0x0) at dh_gen.c:88 > #5 0x00007f4bf2d5730b in XrdCryptosslCipher::XrdCryptosslCipher (this=0x7f4b75f2b060, padded=<optimized out>, bits=512, pub=0x0, lpub=<optimized out>, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslCipher.cc:485 > #6 0x00007f4bf2d63c05 in XrdCryptosslFactory::Cipher (this=<optimized out>, b=1, p=0x0, l=0, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslFactory.cc:219 > #7 0x00007f4bf2f7f11a in XrdSecProtocolgsi::ParseCrypto ***@***.***=0x7f4b6bcae380, clist=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:4974 > #8 0x00007f4bf2f8a9f4 in XrdSecProtocolgsi::ServerDoCertreq (this=0x7f4b6bcae380, br=0x7f4b75eb5da0, bm=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3565 > #9 0x00007f4bf2f8ae15 in XrdSecProtocolgsi::ParseServerInput ***@***.***=0x7f4b6bcae380, ***@***.***=0x7f4b75eb5da0, ***@***.***=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3503 > #10 0x00007f4bf2f8b1d8 in XrdSecProtocolgsi::Authenticate (this=0x7f4b6bcae380, cred=<optimized out>, parms=0x7f4bbb1fd3a8, ei=0x7f4bbb1fd3d0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1816 > #11 0x00007f4bf9ecce74 in XrdXrootdProtocol::do_Auth ***@***.***=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:201 > #12 0x00007f4bf9ebf927 in XrdXrootdProtocol::Process2 (this=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:519 > #13 0x00007f4bf9c03bb6 in XrdLinkXeq::DoIt (this=0x7f4bafe458c8) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:320 > #14 0x00007f4bf9c00389 in XrdLink::setProtocol (this=0x7f4bafe458c8, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:435 > #15 0x00007f4bf9c06c8a in XrdScheduler::Run (this=0x614e60 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:406 > #16 0x00007f4bf9c06da9 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:89 > #17 0x00007f4bf9b94e37 in XrdSysThread_Xeq (myargs=0x7f4b0c6130c0) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86 > #18 0x00007f4bf8cfdea5 in start_thread (arg=0x7f4bbb1fe700) at pthread_create.c:307 > #19 0x00007f4bf8a269fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > ``` > </details> > > When the server gets busy, it gets bogged down with key generation. I assume clients may timeout, then retry, making the load worse. > > As a temporary workaround for our site, I switched to loading pre-generated DH parameters from a file (using 2048-bit at present). After applying the patch to our local redirector, load went back to normal. https://github.com/jthiltges/xrootd/compare/v5.4.0...dhparam > > Regards, > John > > -- > Reply to this email directly or view it on GitHub: > https://github.com/xrootd/xrootd/issues/1556#issuecomment-1013515556 > You are receiving this because you modified the open/close state. > > Message ID: ***@***.***> -- Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/1556#issuecomment-1015000722 You are receiving this because you are subscribed to this thread. Message ID: <[log in to unmask]> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1