 |
 |
On the local redirector at Nebraska, we began seeing odd spikes in system load after the upgrade to v5.4.0. Load was quite high, reaching 1000-5000, with the system being mostly unresponsive. As Brian said, the problem appeared related to SSL negotiation overhead, where the server generates new DH parameters for each connection. <details> <summary>Sample backtrace</summary> ``` Thread 108 (Thread 0x7f4bbb1fe700 (LWP 28534)): #0 BN_mod_word (a=a@entry=0x7f4b0f911c00, w=w@entry=1699) at bn_word.c:102 #1 0x00007f4bf8327431 in probable_prime_dh_safe (ctx=0x7f4b6bc2eac0, rem=0x7f4b2db70a98, padd=0x7f4b2db70a80, bits=511, p=0x7f4b0f911c00) at bn_prime.c:502 #2 BN_generate_prime_ex (ret=0x7f4b0f911c00, bits=bits@entry=512, safe=safe@entry=1, add=add@entry=0x7f4b2db70a80, rem=rem@entry=0x7f4b2db70a98, cb=cb@entry=0x0) at bn_prime.c:186 #3 0x00007f4bf83635e9 in dh_builtin_genparams (ret=0x7f4bdedd1ba0, ret=0x7f4bdedd1ba0, cb=0x0, generator=5, prime_len=512) at dh_gen.c:194 #4 DH_generate_parameters_ex (ret=0x7f4bdedd1ba0, prime_len=512, generator=5, cb=0x0) at dh_gen.c:88 #5 0x00007f4bf2d5730b in XrdCryptosslCipher::XrdCryptosslCipher (this=0x7f4b75f2b060, padded=<optimized out>, bits=512, pub=0x0, lpub=<optimized out>, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslCipher.cc:485 #6 0x00007f4bf2d63c05 in XrdCryptosslFactory::Cipher (this=<optimized out>, b=1, p=0x0, l=0, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslFactory.cc:219 #7 0x00007f4bf2f7f11a in XrdSecProtocolgsi::ParseCrypto (this=this@entry=0x7f4b6bcae380, clist=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:4974 #8 0x00007f4bf2f8a9f4 in XrdSecProtocolgsi::ServerDoCertreq (this=0x7f4b6bcae380, br=0x7f4b75eb5da0, bm=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3565 #9 0x00007f4bf2f8ae15 in XrdSecProtocolgsi::ParseServerInput (this=this@entry=0x7f4b6bcae380, br=br@entry=0x7f4b75eb5da0, bm=bm@entry=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3503 #10 0x00007f4bf2f8b1d8 in XrdSecProtocolgsi::Authenticate (this=0x7f4b6bcae380, cred=<optimized out>, parms=0x7f4bbb1fd3a8, ei=0x7f4bbb1fd3d0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1816 #11 0x00007f4bf9ecce74 in XrdXrootdProtocol::do_Auth (this=this@entry=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:201 #12 0x00007f4bf9ebf927 in XrdXrootdProtocol::Process2 (this=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:519 #13 0x00007f4bf9c03bb6 in XrdLinkXeq::DoIt (this=0x7f4bafe458c8) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:320 #14 0x00007f4bf9c00389 in XrdLink::setProtocol (this=0x7f4bafe458c8, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:435 #15 0x00007f4bf9c06c8a in XrdScheduler::Run (this=0x614e60 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:406 #16 0x00007f4bf9c06da9 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:89 #17 0x00007f4bf9b94e37 in XrdSysThread_Xeq (myargs=0x7f4b0c6130c0) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86 #18 0x00007f4bf8cfdea5 in start_thread (arg=0x7f4bbb1fe700) at pthread_create.c:307 #19 0x00007f4bf8a269fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ``` </details> When the server gets busy, it gets bogged down with key generation. I assume clients may timeout, then retry, making the load worse. As a temporary workaround for our site, I switched to loading pre-generated DH parameters from a file (using 2048-bit at present). After applying the patch to our local redirector, load went back to normal. https://github.com/jthiltges/xrootd/compare/v5.4.0...dhparam Regards, John -- Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/1556#issuecomment-1013515556 You are receiving this because you are subscribed to this thread. Message ID: <[log in to unmask]> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1