Hi JT, Thanks for working on this. Weird that any use of PEM_read_bio would produce this error but not alll of the time. So, something common has to be triggering it. BTW Do your gridftp nodes also run xrootd/https? Andy On Wed, 2 Jun 2021, jthiltges wrote: >> From setting a breakpoint on [PEM_R_NO_START_LINE](https://github.com/openssl/openssl/blob/OpenSSL_1_0_2k/crypto/pem/pem_lib.c#L707), the hits seemed to fall into three groups below. (This is just from glancing, and not a thorough review.) > For next steps, I'm hoping to handle the parsing error that `XrdCryptosslX509ParseBucket()` may cause, and see if that's enough to address the issue, without the `ERR_clear_error()` calls. > > <details> > <summary>xrootd</summary> > > ``` > Breakpoint 1, PEM_read_bio ***@***.***=0x4385490, ***@***.***=0x7f624fd905e8, ***@***.***=0x7f624fd905f0, ***@***.***=0x7f624fd905f8, ***@***.***=0x7f624fd90600) at pem_lib.c:707 > 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE); > > #0 PEM_read_bio ***@***.***=0x4385490, ***@***.***=0x7f624fd905e8, ***@***.***=0x7f624fd905f0, ***@***.***=0x7f624fd905f8, ***@***.***=0x7f624fd90600) at pem_lib.c:707 > #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f624fd90690, ***@***.***=0x7f624fd90698, ***@***.***=0x7f624fd90680, name=<optimized out>, ***@***.***=0x7f625e1ba4cf "ANY PRIVATE KEY", bp=0x4385490, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291 > #2 0x00007f625e12e6ab in PEM_read_bio_PrivateKey (bp=<optimized out>, ***@***.***=0x0, cb=0x0, u=0x0) at pem_pkey.c:88 > #3 0x00007f625e12cd3b in PEM_read_bio_RSAPrivateKey (bp=<optimized out>, rsa=0x7f624fd90b10, cb=<optimized out>, u=<optimized out>) at pem_all.c:175 > #4 0x00007f6259790c1e in XrdCryptosslX509ParseBucket (b=0x4edff00, chain=0x59fa180) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslAux.cc:607 > #5 0x00007f6259bd8ed9 in XrdSecProtocolgsi::ClientDoCert (this=0x4f56000, br=0x45a09a0, bm=0x7f624fd90d40, emsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3145 > #6 0x00007f6259be6ac5 in XrdSecProtocolgsi::ParseClientInput ***@***.***=0x4f56000, ***@***.***=0x45a09a0, ***@***.***=0x7f624fd90d40, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2930 > #7 0x00007f6259be6e79 in XrdSecProtocolgsi::getCredentials (this=0x4f56000, parm=<optimized out>, ei=0x7f624fd91030) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1499 > #8 0x00007f62552b425d in XrdCl::XRootDTransport::DoAuthentication ***@***.***=0x4001c60, ***@***.***=0x4276500, ***@***.***=0x42098c0) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:2156 > #9 0x00007f62552b480e in XrdCl::XRootDTransport::HandShakeMain ***@***.***=0x4001c60, ***@***.***=0x4276500, channelData=...) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:527 > #10 0x00007f62552b499b in XrdCl::XRootDTransport::HandShake (this=0x4001c60, handShakeData=0x4276500, channelData=...) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:414 > #11 0x00007f62553264bf in XrdCl::AsyncSocketHandler::HandleHandShake ***@***.***=0xa337040) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:708 > #12 0x00007f6255326f10 in XrdCl::AsyncSocketHandler::OnReadWhileHandshaking (this=0xa337040) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:695 > #13 0x00007f62553270d5 in XrdCl::AsyncSocketHandler::Event (this=0xa337040, type=1 '\001') at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:246 > #14 0x00007f62552a02d7 in (anonymous namespace)::SocketCallBack::Event (this=0x4af51c0, chP=<optimized out>, cbArg=<optimized out>, evFlags=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClPollerBuiltIn.cc:83 > #15 0x00007f625f8ef5ad in XrdSys::IOEvents::Poller::CbkXeq ***@***.***=0x3a12dd0, ***@***.***=0x41c4640, events=1, eNum=0, eTxt=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEvents.cc:693 > #16 0x00007f625f8f0709 in XrdSys::IOEvents::PollE::Dispatch ***@***.***=0x3a12dd0, cP=0x41c4640, pollEv=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEventsPollE.icc:269 > #17 0x00007f625f8f08e9 in XrdSys::IOEvents::PollE::Begin (this=0x3a12dd0, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEventsPollE.icc:224 > #18 0x00007f625f8ed2fd in XrdSys::IOEvents::BootStrap::Start (parg=0x7f6250f96b90) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEvents.cc:131 > #19 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x23d6e00) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86 > #20 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0 > #21 0x00007f625e7a29fd in clone () from /lib64/libc.so.6 > ``` > </details> > > <details> > <summary>lcmaps globus</summary> > > ``` > Breakpoint 1, PEM_read_bio ***@***.***=0x575e8c0, ***@***.***=0x7f622dd43278, ***@***.***=0x7f622dd43280, ***@***.***=0x7f622dd43288, ***@***.***=0x7f622dd43290) at pem_lib.c:707 > 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE); > > #0 PEM_read_bio ***@***.***=0x575e8c0, ***@***.***=0x7f622dd43278, ***@***.***=0x7f622dd43280, ***@***.***=0x7f622dd43288, ***@***.***=0x7f622dd43290) at pem_lib.c:707 > #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f622dd43318, ***@***.***=0x7f622dd43320, ***@***.***=0x0, name=<optimized out>, ***@***.***=0x7f625e1ba3b8 "TRUSTED CERTIFICATE", ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291 > #2 0x00007f625e12ddbf in PEM_ASN1_read_bio (d2i=0x7f625e117670 <d2i_X509_AUX>, ***@***.***=0x7f625e1ba3b8 "TRUSTED CERTIFICATE", ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_oth.c:78 > #3 0x00007f625e12dbcf in PEM_read_bio_X509_AUX ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_xaux.c:68 > #4 0x00007f625e13a702 in X509_load_cert_file ***@***.***=0x537be00, file=<optimized out>, type=<optimized out>) at by_file.c:145 > #5 0x00007f625e13ae4f in get_cert_by_subject (xl=0x537be00, type=1, name=<optimized out>, ret=0x7f622dd43630) at by_dir.c:369 > #6 0x00007f625e13865c in X509_STORE_get_by_subject ***@***.***=0x570f100, ***@***.***=1, ***@***.***=0x463ea50, ***@***.***=0x7f622dd436a0) at x509_lu.c:313 > #7 0x00007f625e138e8a in X509_STORE_CTX_get1_issuer (issuer=0x7f622dd43750, ctx=0x570f100, x=0x4669110) at x509_lu.c:617 > #8 0x00007f625e134134 in X509_verify_cert (ctx=<optimized out>) at x509_vfy.c:362 > #9 0x00007f625957e971 in globus_verify(x509_st*, stack_st_X509*, char**) () from /lib64/libXrdLcmaps-5.so > #10 0x00007f625957c8d3 in GetKey(x509_st*, stack_st_X509*, XrdSecEntity&) () from /lib64/libXrdLcmaps-5.so > #11 0x00007f6259578e4d in XrdSecgsiAuthzKey () from /lib64/libXrdLcmaps-5.so > #12 0x00007f6259be4c44 in XrdSecProtocolgsi::Authenticate (this=0x56db600, cred=<optimized out>, parms=0x7f622dd443b8, ei=0x7f622dd443e0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2026 > #13 0x00007f625fbdd29b in XrdXrootdProtocol::do_Auth (this=0x23c7000) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:200 > #14 0x00007f625f932049 in XrdLinkXeq::DoIt (this=0x23afb50) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:302 > #15 0x00007f625f92e9f9 in XrdLink::setProtocol (this=0x23afb50, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:425 > #16 0x00007f625f93534f in XrdScheduler::Run (this=0x614c40 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:382 > #17 0x00007f625f935499 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:88 > #18 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x51d8400) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86 > #19 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0 > #20 0x00007f625e7a29fd in clone () from /lib64/libc.so.6 > ``` > </details> > > <details> > <summary>lcmaps voms</summary> > > ``` > Breakpoint 1, PEM_read_bio ***@***.***=0x3c2dc70, ***@***.***=0x7f624bf85348, ***@***.***=0x7f624bf85350, ***@***.***=0x7f624bf85358, ***@***.***=0x7f624bf85360) at pem_lib.c:707 > 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE); > > #0 PEM_read_bio ***@***.***=0x3c2dc70, ***@***.***=0x7f624bf85348, ***@***.***=0x7f624bf85350, ***@***.***=0x7f624bf85358, ***@***.***=0x7f624bf85360) at pem_lib.c:707 > #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f624bf853e8, ***@***.***=0x7f624bf853f0, ***@***.***=0x0, name=<optimized out>, ***@***.***=0x7f625e1ba3cc "X509 CRL", ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291 > #2 0x00007f625e12ddbf in PEM_ASN1_read_bio (d2i=0x7f625e1183d0 <d2i_X509_CRL>, ***@***.***=0x7f625e1ba3cc "X509 CRL", ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_oth.c:78 > #3 0x00007f625e12caaf in PEM_read_bio_X509_CRL ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_all.c:143 > #4 0x00007f625e13a906 in X509_load_crl_file ***@***.***=0x5c29260, file=<optimized out>, type=<optimized out>) at by_file.c:207 > #5 0x00007f625e13af02 in get_cert_by_subject (xl=0x5c29260, type=2, name=<optimized out>, ret=0x7f624bf85700) at by_dir.c:372 > #6 0x00007f625e13865c in X509_STORE_get_by_subject (vs=0x57f7c00, type=2, name=0x4a4a7e0, ret=0x7f624bf857a0) at x509_lu.c:313 > #7 0x00007f625911cbc1 in proxy_verify_callback () from /lib64/libvomsapi.so.1 > #8 0x00007f625e132d49 in internal_verify (ctx=0x57f7c00) at x509_vfy.c:1920 > #9 0x00007f625e134c4f in X509_verify_cert (ctx=<optimized out>) at x509_vfy.c:500 > #10 0x00007f6259100514 in vomsdata::check_cert(stack_st_X509*) () from /lib64/libvomsapi.so.1 > #11 0x00007f62591010cd in vomsdata::check_from_file(ACC*, std::basic_ifstream<char, std::char_traits<char> >&, std::string const&, std::string const&) () from /lib64/libvomsapi.so.1 > #12 0x00007f6259101846 in vomsdata::check(void*) () from /lib64/libvomsapi.so.1 > #13 0x00007f6259102133 in vomsdata::verifydata(ACC*, std::string const&, std::string const&, x509_st*, voms&) () from /lib64/libvomsapi.so.1 > #14 0x00007f6259102567 in vomsdata::evaluate(ACSEQ*, std::string const&, std::string const&, x509_st*) () from /lib64/libvomsapi.so.1 > #15 0x00007f6259104028 in vomsdata::Retrieve(x509_st*, stack_st_X509*, recurse_type) () from /lib64/libvomsapi.so.1 > #16 0x00007f6259111c25 in VOMS_Retrieve () from /lib64/libvomsapi.so.1 > #17 0x00007f625957cbc1 in GetKey(x509_st*, stack_st_X509*, XrdSecEntity&) () from /lib64/libXrdLcmaps-5.so > #18 0x00007f6259578e4d in XrdSecgsiAuthzKey () from /lib64/libXrdLcmaps-5.so > #19 0x00007f6259be4c44 in XrdSecProtocolgsi::Authenticate (this=0x43b4000, cred=<optimized out>, parms=0x7f624bf873b8, ei=0x7f624bf873e0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2026 > #20 0x00007f625fbdd29b in XrdXrootdProtocol::do_Auth (this=0x48d8800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:200 > #21 0x00007f625f932049 in XrdLinkXeq::DoIt (this=0x23abbf8) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:302 > #22 0x00007f625f92e9f9 in XrdLink::setProtocol (this=0x23abbf8, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:425 > #23 0x00007f625f93534f in XrdScheduler::Run (this=0x614c40 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:382 > #24 0x00007f625f935499 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:88 > #25 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x48ae5e0) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86 > #26 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0 > #27 0x00007f625e7a29fd in clone () from /lib64/libc.so.6 > ``` > </details> > > -- > You are receiving this because you modified the open/close state. > Reply to this email directly or view it on GitHub: > https://github.com/xrootd/xrootd/pull/1464#issuecomment-853542297 -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/pull/1464#issuecomment-853545582 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1