Dear Bokjoo, I tried your script, and I can connect without problems to 5.5 servers here at CERN where I have access via gsi protocol. I get permission denied on your server, but no TLS error. I tried both from CentOS 7 (OpenSSL 1.0.x) and from my own machine (where I use OpenSSL 1.1.x) with client v5.6.1. $ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs xroots://eoscms.cern.ch query config version sec_Client: protocol request for host eoscms.cern.ch token='&P=krb5,[log in to unmask]&P=gsi,v:10600,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.+13:/etc/eos.keytab&P=unix' sec_PM: Skipping krb5 only want gsi sec_PM: Loaded gsi protocol object from libXrdSecgsi.so Secgsi ------------------------------------------------------------------- Secgsi Mode: client Secgsi Debug: 1 Secgsi CA dir: /etc/grid-security/certificates/ Secgsi CA verification level: verifyss Secgsi CRL dir: /etc/grid-security/certificates/ Secgsi CRL extension: .r0 Secgsi CRL check level: try Secgsi CRL refresh time: 86400 Secgsi Certificate: /home/amadio/.globus/usercert.pem Secgsi Key: /home/amadio/.globus/userkey.pem Secgsi Proxy file: /tmp/x509up_u75748 Secgsi Proxy validity: 12:00 Secgsi Proxy dep length: 0 Secgsi Proxy bits: 512 Secgsi Proxy sign option: 1 Secgsi Proxy delegation option: 0 Secgsi Pure Cert/Key authentication allowed Secgsi Allowed server names: [*/]<target host name>[/*] Secgsi Crypto modules: ssl Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc Secgsi MDigests: sha256 Secgsi Trusting DNS for hostname checking Secgsi ------------------------------------------------------------------- sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:5168735f.0|4339b4bc.0' 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions 230904 10:30:53 125169 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (5168735f.0) 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 4 extensions 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 12 extensions 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions 5.5.10 Could you please post a verbose output of the error you see, like I did above for the working case? I think that the client might be missing some configuration or some certificates which are required to validate the server and that's why you see errors. With XrdSecDEBUG=1 XrdSecPROTOCOL=gsi we should be able to get enough information to debug this. Best regards, -Guilherme On Tue, Aug 29, 2023 at 12:39:40PM -0400, Bockjoo Kim wrote: > Yes, as I posted in the original email: "If I switch to 5.5.5 client, there is no issue" > Thanks, > Bockjoo > > On 8/29/23 12:11, Brian Lin wrote: > > Hi all, > > > > We're currently seeing similar TLS issues in osg-test with xrdcp. It's > > why we haven't released 5.6.x in the OSG repos yet: > > https://opensciencegrid.atlassian.net/browse/SOFTWARE-5623?focusedCommentId=380033. > > > > I'm also CC'ing Lincoln who is seeing this issue out in the wild with > > a 5.6.1 client against a 5.5.5 server (see attached). For comparison, > > I've also attached a successful copy for a 5.5.5 client vs 5.5.5 > > server for the same file. > > > > Bockjoo: would you be able to try downgrading to a 5.5 client and > > seeing if that resolves your issue? > > > > Thanks, > > Brian > > > > > > On 8/28/23 08:42, Bockjoo Kim wrote: > >> Hi Guilherme, > >> > >> The client machine (xrootd 5.6.1) that I had the issue is a fully > >> credible OSG CE. > >> > >> So, there should be no issue with the CA and X509_CERT_DIR. > >> > >> I have used this python script to reproduce the issue on the client > >> machine: > >> > >> ############################################# > >> > >> import os > >> import sys > >> import errno > >> import subprocess > >> import zlib > >> import random > >> from XRootD import client > >> from XRootD.client.flags import OpenFlags > >> > >> ENDPOINT='cmsio2.rc.ufl.edu:1094' > >> SAM_TEST_FILE='/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root' > >> > >> > >> print ("XRootD Client Versin",client.__version__) > >> cmd = [ "xrdfs "+ENDPOINT+" query config version" ] > >> try: > >> result = subprocess.run(cmd, shell=True, capture_output=True, > >> text=True) > >> print("XRootD Server Version", result.stdout) > >> except subprocess.TimeoutExpired: > >> print("connecting to endpoint timed out") > >> > >> > >> os.environ["X509_CERT_DIR"] = > >> "/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates" > >> os.environ["X509_USER_PROXY"] = "/home/bockjoo/.cmsuser.proxy" > >> os.environ["X509_USER_PROXY_NONCMS"] = "/home/bockjoo/.griduser.proxy" > >> os.environ["XRD_NETWORKSTACK"] = "IPv4" > >> with client.File() as f: > >> status, response = f.open("root://" + ENDPOINT + "/" + \ > >> SAM_TEST_FILE, flags=OpenFlags.READ, timeout=90) > >> if ( not status.ok ): > >> print (("\nopen(root://%s/%s, flags=OpenFlags.READ, > >> time" + \ > >> "out=90)\nXRootDStatus.code=%d \"%s\"\n") % \ > >> (ENDPOINT, SAM_TEST_FILE, status.code, \ > >> status.message.replace("\n", ""))) > >> pass > >> status, data = f.read(offset=0, size=65536, timeout=90) > >> if ( not status.ok ): > >> print(("\n%s\nread(offset=0, size=65536, > >> timeout=90)\n" + \ > >> "XRootDStatus.code=%d \"%s\"\n") % > >> (SAM_TEST_FILE, \ > >> status.code, status.message.replace("\n", > >> ""))) > >> pass > >> print ("Open Status",status.ok) > >> > >> ############################################# > >> > >> You can choose the endpoint and the file of your choosing with the > >> 5.5.5 server > >> > >> to test it. > >> > >> Thanks, > >> > >> Bockjoo > >> > >> On 8/28/23 09:33, Guilherme Amadio wrote: > >>> Dear Bockjoo, > >>> > >>> On Sat, Aug 26, 2023 at 04:14:28PM -0400, Bockjoo Kim wrote: > >>>> Hi, > >>>> > >>>> I am seeing a python XRootD file open issue for the 5.6.1 client > >>>> with a > >>>> 5.5.5 server : > >>>> > >>>> ============================================================= > >>>> > >>>> XRootD Client Versin 5.6.1 > >>>> XRootD Server Version v5.5.5 > >>>> > >>>> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root, > >>>> > >>>> flags=OpenFlags.READ, timeout=90) > >>>> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily > >>>> unavailable: Unable to connect to cmsio2.rc.ufl.edu; error_ssl" > >>>> > >>>> --------------------------------------------------------------------------- > >>>> > >>>> ValueError Traceback (most recent > >>>> call last) > >>>> /tmp/ipykernel_4179061/812350213.py in <module> > >>>> 40 status.message.replace("\n", ""))) > >>>> 41 #pass > >>>> ---> 42 status, data = f.read(offset=0, size=65536, > >>>> timeout=90) > >>>> 43 if ( not status.ok ): > >>>> 44 print(("\n%s\nread(offset=0, size=65536, > >>>> timeout=90)\n" + \ > >>>> > >>>> /opt/cms/services/anaconda3/lib/python3.9/site-packages/XRootD/client/file.py > >>>> > >>>> in read(self, offset, size, timeout, callback) > >>>> 124 return XRootDStatus(self.__file.read(offset, size, > >>>> timeout, callback)) > >>>> 125 > >>>> --> 126 status, response = self.__file.read(offset, size, timeout) > >>>> 127 return XRootDStatus(status), response > >>>> 128 > >>>> > >>>> ValueError: I/O operation on closed file > >>>> > >>>> =============================================================== > >>>> > >>>> Here, XRootD Server is configured with TLS. > >>>> > >>>> If I remove TLS configuration of the 5.5.5 server, there is no issue. > >>>> > >>>> If I switch to 5.5.5 client, there is no issue. > >>>> > >>>> Is this expected? > >>> It may or may not be. When I wrote the patch, I tested several > >>> scenarios > >>> (see > >>> https://github.com/xrootd/xrootd/pull/2031#issuecomment-1589380486). > >>> The error message that you see is likely caused by a client that cannot > >>> validate the server with TLS (because it does not have the proper CA > >>> certificates installed locally). So I suggest you to try with xrdcp > >>> --notlsok option, or export X509_CERT_DIR=/dev/null to force the client > >>> into not being able to do TLS at all. If the directory > >>> /etc/grid-security > >>> exists on your machine, but the client cannot verify the server, and > >>> TLS > >>> is enforced, then this behavior is expected. Otherwise, please export > >>> XRD_LOGLEVEL=Dump, re-run the command and send us the output so I can > >>> investigate this issue further. You may also want to install the proper > >>> certificates to let the client validate the server to be able to use > >>> TLS. > >>> > >>> Best regards, > >>> -Guilherme > >> > >> ######################################################################## > >> Use REPLY-ALL to reply to list > >> > >> To unsubscribe from the XROOTD-L list, click the following link: > >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1