LISTSERV 16.5 - XROOTD-DEV Archives

Hi JT,

Thanks for working on this. Weird that any use of PEM_read_bio would
produce this error but not alll of the time. So, something common has to
be triggering it.

BTW Do your gridftp nodes also run xrootd/https?

Andy

On Wed, 2 Jun 2021, jthiltges wrote:

>> From setting a breakpoint on [PEM_R_NO_START_LINE](https://github.com/openssl/openssl/blob/OpenSSL_1_0_2k/crypto/pem/pem_lib.c#L707), the hits seemed to fall into three groups below. (This is just from glancing, and not a thorough review.)
> For next steps, I'm hoping to handle the parsing error that `XrdCryptosslX509ParseBucket()` may cause, and see if that's enough to address the issue, without the `ERR_clear_error()` calls.
>
> <details>
> <summary>xrootd</summary>
>
> ```
> Breakpoint 1, PEM_read_bio ***@***.***=0x4385490, ***@***.***=0x7f624fd905e8, ***@***.***=0x7f624fd905f0, ***@***.***=0x7f624fd905f8, ***@***.***=0x7f624fd90600) at pem_lib.c:707
> 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE);
>
> #0 PEM_read_bio ***@***.***=0x4385490, ***@***.***=0x7f624fd905e8, ***@***.***=0x7f624fd905f0, ***@***.***=0x7f624fd905f8, ***@***.***=0x7f624fd90600) at pem_lib.c:707
> #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f624fd90690, ***@***.***=0x7f624fd90698, ***@***.***=0x7f624fd90680, name=<optimized out>, ***@***.***=0x7f625e1ba4cf "ANY PRIVATE KEY", bp=0x4385490, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291
> #2 0x00007f625e12e6ab in PEM_read_bio_PrivateKey (bp=<optimized out>, ***@***.***=0x0, cb=0x0, u=0x0) at pem_pkey.c:88
> #3 0x00007f625e12cd3b in PEM_read_bio_RSAPrivateKey (bp=<optimized out>, rsa=0x7f624fd90b10, cb=<optimized out>, u=<optimized out>) at pem_all.c:175
> #4 0x00007f6259790c1e in XrdCryptosslX509ParseBucket (b=0x4edff00, chain=0x59fa180) at /usr/src/debug/xrootd/xrootd/src/XrdCrypto/XrdCryptosslAux.cc:607
> #5 0x00007f6259bd8ed9 in XrdSecProtocolgsi::ClientDoCert (this=0x4f56000, br=0x45a09a0, bm=0x7f624fd90d40, emsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:3145
> #6 0x00007f6259be6ac5 in XrdSecProtocolgsi::ParseClientInput ***@***.***=0x4f56000, ***@***.***=0x45a09a0, ***@***.***=0x7f624fd90d40, cmsg=...) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2930
> #7 0x00007f6259be6e79 in XrdSecProtocolgsi::getCredentials (this=0x4f56000, parm=<optimized out>, ei=0x7f624fd91030) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1499
> #8 0x00007f62552b425d in XrdCl::XRootDTransport::DoAuthentication ***@***.***=0x4001c60, ***@***.***=0x4276500, ***@***.***=0x42098c0) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:2156
> #9 0x00007f62552b480e in XrdCl::XRootDTransport::HandShakeMain ***@***.***=0x4001c60, ***@***.***=0x4276500, channelData=...) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:527
> #10 0x00007f62552b499b in XrdCl::XRootDTransport::HandShake (this=0x4001c60, handShakeData=0x4276500, channelData=...) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClXRootDTransport.cc:414
> #11 0x00007f62553264bf in XrdCl::AsyncSocketHandler::HandleHandShake ***@***.***=0xa337040) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:708
> #12 0x00007f6255326f10 in XrdCl::AsyncSocketHandler::OnReadWhileHandshaking (this=0xa337040) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:695
> #13 0x00007f62553270d5 in XrdCl::AsyncSocketHandler::Event (this=0xa337040, type=1 '\001') at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:246
> #14 0x00007f62552a02d7 in (anonymous namespace)::SocketCallBack::Event (this=0x4af51c0, chP=<optimized out>, cbArg=<optimized out>, evFlags=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdCl/XrdClPollerBuiltIn.cc:83
> #15 0x00007f625f8ef5ad in XrdSys::IOEvents::Poller::CbkXeq ***@***.***=0x3a12dd0, ***@***.***=0x41c4640, events=1, eNum=0, eTxt=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEvents.cc:693
> #16 0x00007f625f8f0709 in XrdSys::IOEvents::PollE::Dispatch ***@***.***=0x3a12dd0, cP=0x41c4640, pollEv=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEventsPollE.icc:269
> #17 0x00007f625f8f08e9 in XrdSys::IOEvents::PollE::Begin (this=0x3a12dd0, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEventsPollE.icc:224
> #18 0x00007f625f8ed2fd in XrdSys::IOEvents::BootStrap::Start (parg=0x7f6250f96b90) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysIOEvents.cc:131
> #19 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x23d6e00) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
> #20 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0
> #21 0x00007f625e7a29fd in clone () from /lib64/libc.so.6
> ```
> </details>
>
> <details>
> <summary>lcmaps globus</summary>
>
> ```
> Breakpoint 1, PEM_read_bio ***@***.***=0x575e8c0, ***@***.***=0x7f622dd43278, ***@***.***=0x7f622dd43280, ***@***.***=0x7f622dd43288, ***@***.***=0x7f622dd43290) at pem_lib.c:707
> 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE);
>
> #0 PEM_read_bio ***@***.***=0x575e8c0, ***@***.***=0x7f622dd43278, ***@***.***=0x7f622dd43280, ***@***.***=0x7f622dd43288, ***@***.***=0x7f622dd43290) at pem_lib.c:707
> #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f622dd43318, ***@***.***=0x7f622dd43320, ***@***.***=0x0, name=<optimized out>, ***@***.***=0x7f625e1ba3b8 "TRUSTED CERTIFICATE", ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291
> #2 0x00007f625e12ddbf in PEM_ASN1_read_bio (d2i=0x7f625e117670 <d2i_X509_AUX>, ***@***.***=0x7f625e1ba3b8 "TRUSTED CERTIFICATE", ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_oth.c:78
> #3 0x00007f625e12dbcf in PEM_read_bio_X509_AUX ***@***.***=0x575e8c0, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_xaux.c:68
> #4 0x00007f625e13a702 in X509_load_cert_file ***@***.***=0x537be00, file=<optimized out>, type=<optimized out>) at by_file.c:145
> #5 0x00007f625e13ae4f in get_cert_by_subject (xl=0x537be00, type=1, name=<optimized out>, ret=0x7f622dd43630) at by_dir.c:369
> #6 0x00007f625e13865c in X509_STORE_get_by_subject ***@***.***=0x570f100, ***@***.***=1, ***@***.***=0x463ea50, ***@***.***=0x7f622dd436a0) at x509_lu.c:313
> #7 0x00007f625e138e8a in X509_STORE_CTX_get1_issuer (issuer=0x7f622dd43750, ctx=0x570f100, x=0x4669110) at x509_lu.c:617
> #8 0x00007f625e134134 in X509_verify_cert (ctx=<optimized out>) at x509_vfy.c:362
> #9 0x00007f625957e971 in globus_verify(x509_st*, stack_st_X509*, char**) () from /lib64/libXrdLcmaps-5.so
> #10 0x00007f625957c8d3 in GetKey(x509_st*, stack_st_X509*, XrdSecEntity&) () from /lib64/libXrdLcmaps-5.so
> #11 0x00007f6259578e4d in XrdSecgsiAuthzKey () from /lib64/libXrdLcmaps-5.so
> #12 0x00007f6259be4c44 in XrdSecProtocolgsi::Authenticate (this=0x56db600, cred=<optimized out>, parms=0x7f622dd443b8, ei=0x7f622dd443e0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2026
> #13 0x00007f625fbdd29b in XrdXrootdProtocol::do_Auth (this=0x23c7000) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:200
> #14 0x00007f625f932049 in XrdLinkXeq::DoIt (this=0x23afb50) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:302
> #15 0x00007f625f92e9f9 in XrdLink::setProtocol (this=0x23afb50, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:425
> #16 0x00007f625f93534f in XrdScheduler::Run (this=0x614c40 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:382
> #17 0x00007f625f935499 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:88
> #18 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x51d8400) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
> #19 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0
> #20 0x00007f625e7a29fd in clone () from /lib64/libc.so.6
> ```
> </details>
>
> <details>
> <summary>lcmaps voms</summary>
>
> ```
> Breakpoint 1, PEM_read_bio ***@***.***=0x3c2dc70, ***@***.***=0x7f624bf85348, ***@***.***=0x7f624bf85350, ***@***.***=0x7f624bf85358, ***@***.***=0x7f624bf85360) at pem_lib.c:707
> 707 PEMerr(PEM_F_PEM_READ_BIO, PEM_R_NO_START_LINE);
>
> #0 PEM_read_bio ***@***.***=0x3c2dc70, ***@***.***=0x7f624bf85348, ***@***.***=0x7f624bf85350, ***@***.***=0x7f624bf85358, ***@***.***=0x7f624bf85360) at pem_lib.c:707
> #1 0x00007f625e12c436 in PEM_bytes_read_bio ***@***.***=0x7f624bf853e8, ***@***.***=0x7f624bf853f0, ***@***.***=0x0, name=<optimized out>, ***@***.***=0x7f625e1ba3cc "X509 CRL", ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0) at pem_lib.c:291
> #2 0x00007f625e12ddbf in PEM_ASN1_read_bio (d2i=0x7f625e1183d0 <d2i_X509_CRL>, ***@***.***=0x7f625e1ba3cc "X509 CRL", ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_oth.c:78
> #3 0x00007f625e12caaf in PEM_read_bio_X509_CRL ***@***.***=0x3c2dc70, ***@***.***=0x0, ***@***.***=0x0, ***@***.***=0x0) at pem_all.c:143
> #4 0x00007f625e13a906 in X509_load_crl_file ***@***.***=0x5c29260, file=<optimized out>, type=<optimized out>) at by_file.c:207
> #5 0x00007f625e13af02 in get_cert_by_subject (xl=0x5c29260, type=2, name=<optimized out>, ret=0x7f624bf85700) at by_dir.c:372
> #6 0x00007f625e13865c in X509_STORE_get_by_subject (vs=0x57f7c00, type=2, name=0x4a4a7e0, ret=0x7f624bf857a0) at x509_lu.c:313
> #7 0x00007f625911cbc1 in proxy_verify_callback () from /lib64/libvomsapi.so.1
> #8 0x00007f625e132d49 in internal_verify (ctx=0x57f7c00) at x509_vfy.c:1920
> #9 0x00007f625e134c4f in X509_verify_cert (ctx=<optimized out>) at x509_vfy.c:500
> #10 0x00007f6259100514 in vomsdata::check_cert(stack_st_X509*) () from /lib64/libvomsapi.so.1
> #11 0x00007f62591010cd in vomsdata::check_from_file(ACC*, std::basic_ifstream<char, std::char_traits<char> >&, std::string const&, std::string const&) () from /lib64/libvomsapi.so.1
> #12 0x00007f6259101846 in vomsdata::check(void*) () from /lib64/libvomsapi.so.1
> #13 0x00007f6259102133 in vomsdata::verifydata(ACC*, std::string const&, std::string const&, x509_st*, voms&) () from /lib64/libvomsapi.so.1
> #14 0x00007f6259102567 in vomsdata::evaluate(ACSEQ*, std::string const&, std::string const&, x509_st*) () from /lib64/libvomsapi.so.1
> #15 0x00007f6259104028 in vomsdata::Retrieve(x509_st*, stack_st_X509*, recurse_type) () from /lib64/libvomsapi.so.1
> #16 0x00007f6259111c25 in VOMS_Retrieve () from /lib64/libvomsapi.so.1
> #17 0x00007f625957cbc1 in GetKey(x509_st*, stack_st_X509*, XrdSecEntity&) () from /lib64/libXrdLcmaps-5.so
> #18 0x00007f6259578e4d in XrdSecgsiAuthzKey () from /lib64/libXrdLcmaps-5.so
> #19 0x00007f6259be4c44 in XrdSecProtocolgsi::Authenticate (this=0x43b4000, cred=<optimized out>, parms=0x7f624bf873b8, ei=0x7f624bf873e0) at /usr/src/debug/xrootd/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:2026
> #20 0x00007f625fbdd29b in XrdXrootdProtocol::do_Auth (this=0x48d8800) at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:200
> #21 0x00007f625f932049 in XrdLinkXeq::DoIt (this=0x23abbf8) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLinkXeq.cc:302
> #22 0x00007f625f92e9f9 in XrdLink::setProtocol (this=0x23abbf8, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:425
> #23 0x00007f625f93534f in XrdScheduler::Run (this=0x614c40 <XrdGlobal::Sched>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:382
> #24 0x00007f625f935499 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:88
> #25 0x00007f625f8f59f7 in XrdSysThread_Xeq (myargs=0x48ae5e0) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
> #26 0x00007f625f4a1ea5 in start_thread () from /lib64/libpthread.so.0
> #27 0x00007f625e7a29fd in clone () from /lib64/libc.so.6
> ```
> </details>
>
> --
> You are receiving this because you modified the open/close state.
> Reply to this email directly or view it on GitHub:
> https://github.com/xrootd/xrootd/pull/1464#issuecomment-853542297

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/xrootd/xrootd/pull/1464#issuecomment-853545582", "url": "https://github.com/xrootd/xrootd/pull/1464#issuecomment-853545582", "name": "View Pull Request" }, "description": "View this Pull Request on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1