Thank you Andy! The error is permanent, I think. Justas, can you please confirm? Matevz On 2/15/22 17:49, Andrew Hanushevsky wrote: > Here are the common problems when the client issues that message: > > 1) The certificate's chain of trust is broken (could be because the root CA > can't be verified, or the root/intermediate certificate has expired). This would > cause sproradic failures (note that if the certificate is improperly installed > then the failure would always occur which is not the case here). > > 2) The certificate must be renewed before the expiry of the certificate to avoid > any conflict arising out of time violation. Ensure that the date/time is set > correctly on your computer since that might be used to assess the validity > period of the SSL certificate of the website. This would cause repated failures > on some sites but not others depending on how close the cert was to expiring. > > 3) The certificate structure is broken, or the certificate's signature can't be > checked. This is the hardest to solve but one reason could me random memory > corruption. I would suggest runningthe client with valgrind and try to match the > valgring log with the invalid cert message. > > 4) Firewall issues may cause problems with they have been enabled for > "encrypted/SSL scanning or checking." > > 5) If tghe server is using only SHA-1 encryption then those cert are flagged as > insecure and need to update their security certificates. However, would most > likely cause permanent errors. > > Andy > > > > On Tue, 15 Feb 2022, Matevz Tadel wrote: > >> Andy, >> >> What's with the invalid certificate in the client log? >> >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport ] >> [transfer-9.ultralight.org:1095.0] Sending out kXR_login request, username: >> root, cgi: >> ?xrd.cc=us&xrd.tz=-8&xrd.appname=xrdcp&xrd.info=&xrd.hostname=xrd-cache-3.ultralight.org&xrd.rn=v5.2.0, >> dual-stack: true, private IPv4: false, private IPv6: false >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock ] >> [transfer-9.ultralight.org:1095.0] TLS hand-shake exchange. >> >> ===> HERE: >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg ] [TLS_Context:] >> Unable to create TLS context; invalid certificate. >> >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock ] >> [transfer-9.ultralight.org:1095.0] Socket error while handshaking: [FATAL] TLS >> error >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock ] >> [transfer-9.ultralight.org:1095.0] Closing the socket >> >> Can I run in gdb to get more info? What is good place to start poking? >> >> I was assuming it's the server cert that client does not like ... but it does >> look ok to me :) >> >> Matevz >> >> >> On 2/15/22 14:04, Andrew Hanushevsky wrote: >>> Hi Bockjoo, >>> >>> Unfortunately, that's not the way it works. While gsi doesn't need to use TLS >>> ztn does. Since the erver doesn't know which protocol the client will eventually >>> settle on, the connection has to use TLS right from the start. That means you >>> cannot use ztn with incapable clients. >>> >>> Andy >>> >>> >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote: >>> >>>> Hi Andy, >>>> >>>> There are two sec.protocols: gsi and ztn. >>>> >>>> Doesn't the interaction go through gsi and if it fails, will it go through ztn? >>>> >>>> For incapable clients, the gsi can succeed, no? >>>> >>>> Thanks, >>>> >>>> Bockjoo >>>> >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote: >>>>> Hi Justas, >>>>> >>>>> If you look into the log you will notice a warning that tells you that TLS >>>>> will always be on regardless of the "capable" setting because authentication >>>>> protocol ztn requires tls. So, this may be the source of the problem, >>>>> certainly it will be for incapable clients. >>>>> >>>>> Andy >>>>> >>>>> >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out 1 server >>>>>> from prod and played with full debug mode, on/off tls. >>>>>> >>>>>> Logs from server/client are available here: >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= >>>>>> >>>>>> >>>>>> To turn TLS, I added this config: >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem >>>>>> /etc/grid-security/xrootd/xrootdkey.pem >>>>>> xrd.tlsca certdir /etc/grid-security/certificates >>>>>> xrootd.tls capable all >>>>>> sec.protocol /usr/lib64 ztn >>>>>> >>>>>> And with TLS on - I always get: >>>>>> TLS hand-shake exchange. >>>>>> Socket error while handshaking: [FATAL] TLS error >>>>>> Closing the socket >>>>>> >>>>>> >>>>>> If it helps, here is full config: >>>>>> >>>>>> all.export /tmp stage >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN >>>>>> all.adminpath /var/spool/xrootd >>>>>> all.pidpath /var/run/xrootd >>>>>> >>>>>> # XrootD Security >>>>>> # --------------------------------------- >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so >>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 >>>>>> -authzfun:libXrdLcmaps.so -authzto:900 >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0 >>>>>> acc.authdb /etc/xrootd/auth_file_stageout >>>>>> ofs.authorize >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret >>>>>> ofs.authlib ++ libXrdMacaroons.so >>>>>> ofs.authlib ++ libXrdAccSciTokens.so >>>>>> # -------------------------------------- >>>>>> # XrootD Monitoring >>>>>> # -------------------------------------- >>>>>> # Monitoring for AAA Dashboard : >>>>>> xrd.report 169.228.130.91:9931 every 30s all sync >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest >>>>>> files io info user 169.228.130.91:9930 dest fstat info user >>>>>> xrd-mon.osgstorage.org:9930 >>>>>> all.sitename T2_US_Caltech >>>>>> # ------------------------------------- >>>>>> # Configure redirector/server >>>>>> # ------------------------------------- >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org >>>>>> xrd.port 1095 >>>>>> all.manager $(xrdr):1213 >>>>>> >>>>>> if $(xrdr) >>>>>> # It's role is manager >>>>>> all.role manager >>>>>> # Redirect all lookup calls to original data servers. Redirector does not >>>>>> have visibility of FS >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed >>>>>> else >>>>>> # Role is server >>>>>> all.role server >>>>>> # The known managers (local redirector) >>>>>> all.manager meta $(xrdr):1213 >>>>>> >>>>>> # Enable xrootd checksum calculation "on-the-fly" using multiuser plugin >>>>>> # This makes XRootD to write the files with the >>>>>> # ownership of the user that authenticated to the server and not as the >>>>>> # 'xrootd' user >>>>>> ofs.osslib ++ libXrdMultiuser.so >>>>>> # Enable the checksum wrapper >>>>>> ofs.ckslib * libXrdMultiuser.so >>>>>> # Control of checksum >>>>>> xrootd.chksum max 10 adler32 >>>>>> multiuser.checksumonwrite on >>>>>> multiuser.umask 0002 >>>>>> >>>>>> fi >>>>>> # ------------------------------------- >>>>>> # Allow only specific path, checksum config >>>>>> # ------------------------------------- >>>>>> # Allow any path to be exported; this is further refined in the authfile. >>>>>> all.export / >>>>>> >>>>>> # Hosts allowed to use this xrootd cluster >>>>>> cms.allow host * >>>>>> >>>>>> # Enable xrootd debugging >>>>>> xrootd.trace emsg login stall redirect >>>>>> cms.trace defer files forward redirect >>>>>> >>>>>> # Disable async. Related issue: >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= >>>>>> >>>>>> xrootd.async off >>>>>> >>>>>> # ------------------------------------- >>>>>> # Integrate with CMS Namespaces >>>>>> # It will see files under /store/... >>>>>> # ------------------------------------- >>>>>> oss.localroot /storage/cms >>>>>> # ------------------------------------- >>>>>> # Configure davs/https for TPC >>>>>> # ------------------------------------- >>>>>> # Enable https over XrootD >>>>>> if exec xrootd >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so >>>>>> http.cadir /etc/grid-security/certificates >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so >>>>>> http.secretkey XXXXXXX >>>>>> # Enable third-party-copy >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so >>>>>> # Pass the bearer token to the Xrootd authorization framework. >>>>>> http.header2cgi Authorization authz >>>>>> http.listingdeny yes >>>>>> http.desthttps yes >>>>>> http.selfhttps2http no >>>>>> http.staticpreload >>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= >>>>>> >>>>>> /etc/xrootd/robots.txt >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so >>>>>> fi >>>>>> >>>>>> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem >>>>>> /etc/grid-security/xrootd/xrootdkey.pem >>>>>> xrd.tlsca certdir /etc/grid-security/certificates >>>>>> xrootd.tls capable all >>>>>> sec.protocol /usr/lib64 ztn >>>>>> >>>>>> >>>>>> >>>>>> xrootd.trace all >>>>>> xrd.trace all >>>>>> ofs.trace all >>>>>> pfc.trace all >>>>>> cms.trace all >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2 >>>>>> Warning, 1 Info) >>>>>> pss.setopt DebugLevel 4 >>>>>> >>>>>> ######################################################################## >>>>>> Use REPLY-ALL to reply to list >>>>>> >>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >>>>>> >>>>>> >>>>> >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >>>>> >>>> >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >>>> >>>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >>> >>> >> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1