Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
On the local redirector at Nebraska, we began seeing odd spikes in system load after the upgrade to v5.4.0. Load was quite high, reaching 1000-5000, with the system being mostly unresponsive.
As Brian said, the problem appeared related to SSL negotiation overhead, where the server generates new DH parameters for each connection.
Thread 108 (Thread 0x7f4bbb1fe700 (LWP 28534)):
#0 BN_mod_word (a=a@entry=0x7f4b0f911c00, w=w@entry=1699) at bn_word.c:102
#1 0x00007f4bf8327431 in probable_prime_dh_safe (ctx=0x7f4b6bc2eac0, rem=0x7f4b2db70a98, padd=0x7f4b2db70a80, bits=511, p=0x7f4b0f911c00) at bn_prime.c:502
#2 BN_generate_prime_ex (ret=0x7f4b0f911c00, bits=bits@entry=512, safe=safe@entry=1, add=add@entry=0x7f4b2db70a80, rem=rem@entry=0x7f4b2db70a98, cb=cb@entry=0x0) at bn_prime.c:186
#3 0x00007f4bf83635e9 in dh_builtin_genparams (ret=0x7f4bdedd1ba0, ret=0x7f4bdedd1ba0, cb=0x0, generator=5, prime_len=512) at dh_gen.c:194
#4 DH_generate_parameters_ex (ret=0x7f4bdedd1ba0, prime_len=512, generator=5, cb=0x0) at dh_gen.c:88
#5 0x00007f4bf2d5730b in XrdCryptosslCipher::XrdCryptosslCipher (this=0x7f4b75f2b060, padded=<optimized out>, bits=512, pub=0x0, lpub=<optimized out>, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslCipher.cc:485
#6 0x00007f4bf2d63c05 in XrdCryptosslFactory::Cipher (this=<optimized out>, b=1, p=0x0, l=0, t=0x0) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslFactory.cc:219
#7 0x00007f4bf2f7f11a in XrdSecProtocolgsi::ParseCrypto (this=this@entry=0x7f4b6bcae380, clist=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:4974
#8 0x00007f4bf2f8a9f4 in XrdSecProtocolgsi::ServerDoCertreq (this=0x7f4b6bcae380, br=0x7f4b75eb5da0, bm=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3565
#9 0x00007f4bf2f8ae15 in XrdSecProtocolgsi::ParseServerInput (this=this@entry=0x7f4b6bcae380, br=br@entry=0x7f4b75eb5da0, bm=bm@entry=0x7f4bbb1fd168, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3503
#10 0x00007f4bf2f8b1d8 in XrdSecProtocolgsi::Authenticate (this=0x7f4b6bcae380, cred=<optimized out>, parms=0x7f4bbb1fd3a8, ei=0x7f4bbb1fd3d0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1816
#11 0x00007f4bf9ecce74 in XrdXrootdProtocol::do_Auth (this=this@entry=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:201
#12 0x00007f4bf9ebf927 in XrdXrootdProtocol::Process2 (this=0x7f4bbf75c800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:519
#13 0x00007f4bf9c03bb6 in XrdLinkXeq::DoIt (this=0x7f4bafe458c8) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:320
#14 0x00007f4bf9c00389 in XrdLink::setProtocol (this=0x7f4bafe458c8, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:435
#15 0x00007f4bf9c06c8a in XrdScheduler::Run (this=0x614e60 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:406
#16 0x00007f4bf9c06da9 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:89
#17 0x00007f4bf9b94e37 in XrdSysThread_Xeq (myargs=0x7f4b0c6130c0) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
#18 0x00007f4bf8cfdea5 in start_thread (arg=0x7f4bbb1fe700) at pthread_create.c:307
#19 0x00007f4bf8a269fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
When the server gets busy, it gets bogged down with key generation. I assume clients may timeout, then retry, making the load worse.
As a temporary workaround for our site, I switched to loading pre-generated DH parameters from a file (using 2048-bit at present). After applying the patch to our local redirector, load went back to normal. jthiltges/xrootd@v5.4.0...dhparam
Regards,
John
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1