Thank you Andy!

The error is permanent, I think. Justas, can you please confirm?

Matevz

On 2/15/22 17:49, Andrew Hanushevsky wrote:
> Here are the common problems when the client issues that message:
>
> 1) The certificate's chain of trust is broken (could be because the root CA
> can't be verified, or the root/intermediate certificate has expired). This would
> cause sproradic failures (note that if the certificate is improperly installed
> then the failure would always occur which is not the case here).
>
> 2) The certificate must be renewed before the expiry of the certificate to avoid
> any conflict arising out of time violation. Ensure that the date/time is set
> correctly on your computer since that might be used to assess the validity
> period of the SSL certificate of the website. This would cause repated failures
> on some sites but not others depending on how close the cert was to expiring.
>
> 3) The certificate structure is broken, or the certificate's signature can't be
> checked. This is the hardest to solve but one reason could me random memory
> corruption. I would suggest runningthe client with valgrind and try to match the
> valgring log with the invalid cert message.
>
> 4) Firewall issues may cause problems with they have been enabled for
> "encrypted/SSL scanning or checking."
>
> 5) If tghe server is using only SHA-1 encryption then those cert are flagged as
> insecure and need to update their security certificates. However, would most
> likely cause permanent errors.
>
> Andy
>
>
>
> On Tue, 15 Feb 2022, Matevz Tadel wrote:
>
>> Andy,
>>
>> What's with the invalid certificate in the client log?
>>
>> [2022-02-15 09:56:19.549139 -0800][Debug  ][XRootDTransport   ]
>> [transfer-9.ultralight.org:1095.0] Sending out kXR_login request, username:
>> root, cgi:
>> ?xrd.cc=us&xrd.tz=-8&xrd.appname=xrdcp&xrd.info=&xrd.hostname=xrd-cache-3.ultralight.org&xrd.rn=v5.2.0,
>> dual-stack: true, private IPv4: false, private IPv6: false
>> [2022-02-15 09:56:19.549209 -0800][Debug  ][AsyncSock         ]
>> [transfer-9.ultralight.org:1095.0] TLS hand-shake exchange.
>>
>> ===> HERE:
>> [2022-02-15 09:56:19.551762 -0800][Error  ][TlsMsg            ] [TLS_Context:]
>> Unable to create TLS context; invalid certificate.
>>
>> [2022-02-15 09:56:19.551903 -0800][Error  ][AsyncSock         ]
>> [transfer-9.ultralight.org:1095.0] Socket error while handshaking: [FATAL] TLS
>> error
>> [2022-02-15 09:56:19.551920 -0800][Debug  ][AsyncSock         ]
>> [transfer-9.ultralight.org:1095.0] Closing the socket
>>
>> Can I run in gdb to get more info? What is good place to start poking?
>>
>> I was assuming it's the server cert that client does not like ... but it does
>> look ok to me :)
>>
>> Matevz
>>
>>
>> On 2/15/22 14:04, Andrew Hanushevsky wrote:
>>> Hi Bockjoo,
>>>
>>> Unfortunately, that's not the way it works. While gsi doesn't need to use TLS
>>> ztn does. Since the erver doesn't know which protocol the client will eventually
>>> settle on, the connection has to use TLS right from the start. That means you
>>> cannot use ztn with incapable clients.
>>>
>>> Andy
>>>
>>>
>>> On Tue, 15 Feb 2022, Bockjoo Kim wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> There are two sec.protocols: gsi and ztn.
>>>>
>>>> Doesn't the interaction go through gsi and if it fails, will it go through ztn?
>>>>
>>>> For incapable clients, the gsi can succeed, no?
>>>>
>>>> Thanks,
>>>>
>>>> Bockjoo
>>>>
>>>> On 2/15/22 14:52, Andrew Hanushevsky wrote:
>>>>> Hi Justas,
>>>>>
>>>>> If you look into the log you will notice a warning that tells you that TLS
>>>>> will always be on regardless of the "capable" setting because authentication
>>>>> protocol ztn requires tls. So, this may be the source of the problem,
>>>>> certainly it will be for incapable clients.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>> On Tue, 15 Feb 2022, Justas Balcas wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out 1 server
>>>>>> from prod and played with full debug mode, on/off tls.
>>>>>>
>>>>>> Logs from server/client are available here:
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e=
>>>>>>
>>>>>>
>>>>>> To turn TLS, I added this config:
>>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem
>>>>>> /etc/grid-security/xrootd/xrootdkey.pem
>>>>>> xrd.tlsca certdir /etc/grid-security/certificates
>>>>>> xrootd.tls capable all
>>>>>> sec.protocol /usr/lib64 ztn
>>>>>>
>>>>>> And with TLS on - I always get:
>>>>>> TLS hand-shake exchange.
>>>>>> Socket error while handshaking: [FATAL] TLS error
>>>>>> Closing the socket
>>>>>>
>>>>>>
>>>>>> If it helps, here is full config:
>>>>>>
>>>>>> all.export /tmp stage
>>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN
>>>>>> all.adminpath /var/spool/xrootd
>>>>>> all.pidpath /var/run/xrootd
>>>>>>
>>>>>> # XrootD Security
>>>>>> # ---------------------------------------
>>>>>> xrootd.seclib /usr/lib64/libXrdSec.so
>>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates
>>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem
>>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3
>>>>>> -authzfun:libXrdLcmaps.so -authzto:900
>>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0
>>>>>> acc.authdb /etc/xrootd/auth_file_stageout
>>>>>> ofs.authorize
>>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret
>>>>>> ofs.authlib ++ libXrdMacaroons.so
>>>>>> ofs.authlib ++ libXrdAccSciTokens.so
>>>>>> # --------------------------------------
>>>>>> # XrootD Monitoring
>>>>>> # --------------------------------------
>>>>>> # Monitoring for AAA Dashboard :
>>>>>> xrd.report 169.228.130.91:9931 every 30s all sync
>>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest
>>>>>> files io info user 169.228.130.91:9930 dest fstat info user
>>>>>> xrd-mon.osgstorage.org:9930
>>>>>> all.sitename T2_US_Caltech
>>>>>> # -------------------------------------
>>>>>> # Configure redirector/server
>>>>>> # -------------------------------------
>>>>>> set xrdr = xrootd-redir-stageout.ultralight.org
>>>>>> xrd.port 1095
>>>>>> all.manager $(xrdr):1213
>>>>>>
>>>>>> if $(xrdr)
>>>>>>  # It's role is manager
>>>>>>  all.role manager
>>>>>>  # Redirect all lookup calls to original data servers. Redirector does not
>>>>>> have visibility of FS
>>>>>>  cms.dfs lookup distrib mdhold 20m redirect immed
>>>>>> else
>>>>>>  # Role is server
>>>>>>  all.role server
>>>>>>  # The known managers (local redirector)
>>>>>>  all.manager meta $(xrdr):1213
>>>>>>
>>>>>>  # Enable xrootd checksum calculation "on-the-fly" using multiuser plugin
>>>>>>  # This makes XRootD to write the files with the
>>>>>>  # ownership of the user that authenticated to the server and not as the
>>>>>>  # 'xrootd' user
>>>>>>  ofs.osslib ++ libXrdMultiuser.so
>>>>>>  # Enable the checksum wrapper
>>>>>>  ofs.ckslib * libXrdMultiuser.so
>>>>>>  # Control of checksum
>>>>>>  xrootd.chksum max 10 adler32
>>>>>>  multiuser.checksumonwrite on
>>>>>>  multiuser.umask 0002
>>>>>>
>>>>>> fi
>>>>>> # -------------------------------------
>>>>>> # Allow only specific path, checksum config
>>>>>> # -------------------------------------
>>>>>> # Allow any path to be exported; this is further refined in the authfile.
>>>>>> all.export /
>>>>>>
>>>>>> # Hosts allowed to use this xrootd cluster
>>>>>> cms.allow host *
>>>>>>
>>>>>> # Enable xrootd debugging
>>>>>> xrootd.trace emsg login stall redirect
>>>>>> cms.trace defer files forward redirect
>>>>>>
>>>>>> # Disable async. Related issue:
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e=
>>>>>>
>>>>>> xrootd.async off
>>>>>>
>>>>>> # -------------------------------------
>>>>>> # Integrate with CMS Namespaces
>>>>>> # It will see files under /store/...
>>>>>> # -------------------------------------
>>>>>> oss.localroot /storage/cms
>>>>>> # -------------------------------------
>>>>>> # Configure davs/https for TPC
>>>>>> # -------------------------------------
>>>>>> # Enable https over XrootD
>>>>>> if exec xrootd
>>>>>>  xrd.protocol http:1095 /usr/lib64/libXrdHttp.so
>>>>>>  http.cadir /etc/grid-security/certificates
>>>>>>  http.cert /etc/grid-security/xrootd/xrootdcert.pem
>>>>>>  http.key /etc/grid-security/xrootd/xrootdkey.pem
>>>>>>  http.secxtractor /usr/lib64/libXrdLcmaps.so
>>>>>>  http.secretkey XXXXXXX
>>>>>>  # Enable third-party-copy
>>>>>>  http.exthandler xrdtpc libXrdHttpTPC.so
>>>>>>  # Pass the bearer token to the Xrootd authorization framework.
>>>>>>  http.header2cgi Authorization authz
>>>>>>  http.listingdeny yes
>>>>>>  http.desthttps yes
>>>>>>  http.selfhttps2http no
>>>>>>  http.staticpreload
>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= 
>>>>>>
>>>>>> /etc/xrootd/robots.txt
>>>>>>  http.exthandler xrdmacaroons libXrdMacaroons.so
>>>>>> fi
>>>>>>
>>>>>>
>>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem
>>>>>> /etc/grid-security/xrootd/xrootdkey.pem
>>>>>> xrd.tlsca certdir /etc/grid-security/certificates
>>>>>> xrootd.tls capable all
>>>>>> sec.protocol /usr/lib64 ztn
>>>>>>
>>>>>>
>>>>>>
>>>>>> xrootd.trace all
>>>>>> xrd.trace    all
>>>>>> ofs.trace    all
>>>>>> pfc.trace    all
>>>>>> cms.trace    all
>>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2
>>>>>> Warning, 1 Info)
>>>>>> pss.setopt   DebugLevel 4
>>>>>>
>>>>>> ########################################################################
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>>>>>>
>>>>>>
>>>>>
>>>>> ########################################################################
>>>>> Use REPLY-ALL to reply to list
>>>>>
>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>>>>>
>>>>
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>>>>
>>>>
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>>>
>>>
>>