 |
 |
That is correct, Matevz On Tue, Feb 15, 2022, 18:39 Matevz Tadel <[log in to unmask]> wrote: > Thank you Andy! > > The error is permanent, I think. Justas, can you please confirm? > > Matevz > > On 2/15/22 17:49, Andrew Hanushevsky wrote: > > Here are the common problems when the client issues that message: > > > > 1) The certificate's chain of trust is broken (could be because the root > CA > > can't be verified, or the root/intermediate certificate has expired). > This would > > cause sproradic failures (note that if the certificate is improperly > installed > > then the failure would always occur which is not the case here). > > > > 2) The certificate must be renewed before the expiry of the certificate > to avoid > > any conflict arising out of time violation. Ensure that the date/time is > set > > correctly on your computer since that might be used to assess the > validity > > period of the SSL certificate of the website. This would cause repated > failures > > on some sites but not others depending on how close the cert was to > expiring. > > > > 3) The certificate structure is broken, or the certificate's signature > can't be > > checked. This is the hardest to solve but one reason could me random > memory > > corruption. I would suggest runningthe client with valgrind and try to > match the > > valgring log with the invalid cert message. > > > > 4) Firewall issues may cause problems with they have been enabled for > > "encrypted/SSL scanning or checking." > > > > 5) If tghe server is using only SHA-1 encryption then those cert are > flagged as > > insecure and need to update their security certificates. However, would > most > > likely cause permanent errors. > > > > Andy > > > > > > > > On Tue, 15 Feb 2022, Matevz Tadel wrote: > > > >> Andy, > >> > >> What's with the invalid certificate in the client log? > >> > >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport ] > >> [transfer-9.ultralight.org:1095.0] Sending out kXR_login request, > username: > >> root, cgi: > >> ?xrd.cc=us&xrd.tz=-8&xrd.appname=xrdcp&xrd.info=&xrd.hostname= > xrd-cache-3.ultralight.org&xrd.rn=v5.2.0, > >> dual-stack: true, private IPv4: false, private IPv6: false > >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock ] > >> [transfer-9.ultralight.org:1095.0] TLS hand-shake exchange. > >> > >> ===> HERE: > >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg ] > [TLS_Context:] > >> Unable to create TLS context; invalid certificate. > >> > >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock ] > >> [transfer-9.ultralight.org:1095.0] Socket error while handshaking: > [FATAL] TLS > >> error > >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock ] > >> [transfer-9.ultralight.org:1095.0] Closing the socket > >> > >> Can I run in gdb to get more info? What is good place to start poking? > >> > >> I was assuming it's the server cert that client does not like ... but > it does > >> look ok to me :) > >> > >> Matevz > >> > >> > >> On 2/15/22 14:04, Andrew Hanushevsky wrote: > >>> Hi Bockjoo, > >>> > >>> Unfortunately, that's not the way it works. While gsi doesn't need to > use TLS > >>> ztn does. Since the erver doesn't know which protocol the client will > eventually > >>> settle on, the connection has to use TLS right from the start. That > means you > >>> cannot use ztn with incapable clients. > >>> > >>> Andy > >>> > >>> > >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote: > >>> > >>>> Hi Andy, > >>>> > >>>> There are two sec.protocols: gsi and ztn. > >>>> > >>>> Doesn't the interaction go through gsi and if it fails, will it go > through ztn? > >>>> > >>>> For incapable clients, the gsi can succeed, no? > >>>> > >>>> Thanks, > >>>> > >>>> Bockjoo > >>>> > >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote: > >>>>> Hi Justas, > >>>>> > >>>>> If you look into the log you will notice a warning that tells you > that TLS > >>>>> will always be on regardless of the "capable" setting because > authentication > >>>>> protocol ztn requires tls. So, this may be the source of the problem, > >>>>> certainly it will be for incapable clients. > >>>>> > >>>>> Andy > >>>>> > >>>>> > >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out > 1 server > >>>>>> from prod and played with full debug mode, on/off tls. > >>>>>> > >>>>>> Logs from server/client are available here: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= > >>>>>> > >>>>>> > >>>>>> To turn TLS, I added this config: > >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >>>>>> xrootd.tls capable all > >>>>>> sec.protocol /usr/lib64 ztn > >>>>>> > >>>>>> And with TLS on - I always get: > >>>>>> TLS hand-shake exchange. > >>>>>> Socket error while handshaking: [FATAL] TLS error > >>>>>> Closing the socket > >>>>>> > >>>>>> > >>>>>> If it helps, here is full config: > >>>>>> > >>>>>> all.export /tmp stage > >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN > >>>>>> all.adminpath /var/spool/xrootd > >>>>>> all.pidpath /var/run/xrootd > >>>>>> > >>>>>> # XrootD Security > >>>>>> # --------------------------------------- > >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so > >>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates > >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem > >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 > >>>>>> -authzfun:libXrdLcmaps.so -authzto:900 > >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 > -gmapto:0 > >>>>>> acc.authdb /etc/xrootd/auth_file_stageout > >>>>>> ofs.authorize > >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret > >>>>>> ofs.authlib ++ libXrdMacaroons.so > >>>>>> ofs.authlib ++ libXrdAccSciTokens.so > >>>>>> # -------------------------------------- > >>>>>> # XrootD Monitoring > >>>>>> # -------------------------------------- > >>>>>> # Monitoring for AAA Dashboard : > >>>>>> xrd.report 169.228.130.91:9931 every 30s all sync > >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 > dest > >>>>>> files io info user 169.228.130.91:9930 dest fstat info user > >>>>>> xrd-mon.osgstorage.org:9930 > >>>>>> all.sitename T2_US_Caltech > >>>>>> # ------------------------------------- > >>>>>> # Configure redirector/server > >>>>>> # ------------------------------------- > >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org > >>>>>> xrd.port 1095 > >>>>>> all.manager $(xrdr):1213 > >>>>>> > >>>>>> if $(xrdr) > >>>>>> # It's role is manager > >>>>>> all.role manager > >>>>>> # Redirect all lookup calls to original data servers. Redirector > does not > >>>>>> have visibility of FS > >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed > >>>>>> else > >>>>>> # Role is server > >>>>>> all.role server > >>>>>> # The known managers (local redirector) > >>>>>> all.manager meta $(xrdr):1213 > >>>>>> > >>>>>> # Enable xrootd checksum calculation "on-the-fly" using multiuser > plugin > >>>>>> # This makes XRootD to write the files with the > >>>>>> # ownership of the user that authenticated to the server and not > as the > >>>>>> # 'xrootd' user > >>>>>> ofs.osslib ++ libXrdMultiuser.so > >>>>>> # Enable the checksum wrapper > >>>>>> ofs.ckslib * libXrdMultiuser.so > >>>>>> # Control of checksum > >>>>>> xrootd.chksum max 10 adler32 > >>>>>> multiuser.checksumonwrite on > >>>>>> multiuser.umask 0002 > >>>>>> > >>>>>> fi > >>>>>> # ------------------------------------- > >>>>>> # Allow only specific path, checksum config > >>>>>> # ------------------------------------- > >>>>>> # Allow any path to be exported; this is further refined in the > authfile. > >>>>>> all.export / > >>>>>> > >>>>>> # Hosts allowed to use this xrootd cluster > >>>>>> cms.allow host * > >>>>>> > >>>>>> # Enable xrootd debugging > >>>>>> xrootd.trace emsg login stall redirect > >>>>>> cms.trace defer files forward redirect > >>>>>> > >>>>>> # Disable async. Related issue: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= > >>>>>> > >>>>>> xrootd.async off > >>>>>> > >>>>>> # ------------------------------------- > >>>>>> # Integrate with CMS Namespaces > >>>>>> # It will see files under /store/... > >>>>>> # ------------------------------------- > >>>>>> oss.localroot /storage/cms > >>>>>> # ------------------------------------- > >>>>>> # Configure davs/https for TPC > >>>>>> # ------------------------------------- > >>>>>> # Enable https over XrootD > >>>>>> if exec xrootd > >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so > >>>>>> http.cadir /etc/grid-security/certificates > >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so > >>>>>> http.secretkey XXXXXXX > >>>>>> # Enable third-party-copy > >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so > >>>>>> # Pass the bearer token to the Xrootd authorization framework. > >>>>>> http.header2cgi Authorization authz > >>>>>> http.listingdeny yes > >>>>>> http.desthttps yes > >>>>>> http.selfhttps2http no > >>>>>> http.staticpreload > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= > > >>>>>> > >>>>>> /etc/xrootd/robots.txt > >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so > >>>>>> fi > >>>>>> > >>>>>> > >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >>>>>> xrootd.tls capable all > >>>>>> sec.protocol /usr/lib64 ztn > >>>>>> > >>>>>> > >>>>>> > >>>>>> xrootd.trace all > >>>>>> xrd.trace all > >>>>>> ofs.trace all > >>>>>> pfc.trace all > >>>>>> cms.trace all > >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2 > >>>>>> Warning, 1 Info) > >>>>>> pss.setopt DebugLevel 4 > >>>>>> > >>>>>> > ######################################################################## > >>>>>> Use REPLY-ALL to reply to list > >>>>>> > >>>>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >>>>>> > >>>>>> > >>>>> > >>>>> > ######################################################################## > >>>>> Use REPLY-ALL to reply to list > >>>>> > >>>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >>>>> > >>>> > >>>> > >>>> > ######################################################################## > >>>> Use REPLY-ALL to reply to list > >>>> > >>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >>>> > >>>> > >>> > >>> > ######################################################################## > >>> Use REPLY-ALL to reply to list > >>> > >>> To unsubscribe from the XROOTD-L list, click the following link: > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >>> > >>> > >> > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1