Hi Guilherme, Indeed, if I replace 'root:' protocol with 'roots:' protocol, it works. Thanks, Bockjoo On 9/5/23 10:52, Guilherme Amadio wrote: > Dear Bokjoo, > > I think I've finally managed to reproduce this problem. The output log I > get is attached. In it, I see: > > [2023-09-05 16:34:03.004973 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending out kXR_login request, username: amadio, cgi: xrd.cc=ch&xrd.tz=1&xrd.appname=python3.11&xrd.info=&xrd.hostname=gentoo.cern.ch&xrd.rn=v5.6.1, dual-stack: true, private IPv4: false, private IPv6: false > 2023-09-05 16:34:03.004985 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange. > [2023-09-05 16:34:03.120274 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange. > [2023-09-05 16:34:03.120329 +0200][Error ][TlsMsg ] [] TLS error rc=-1 ec=1 (error_ssl) errno=104. > [2023-09-05 16:34:03.120342 +0200][Debug ][TlsMsg ] [] 140201906132672:error:1408F0C6:SSL routines:ssl3_get_record:packet length too long:../openssl-1.1.1v/ssl/record/ssl3_record.c:362: > [2023-09-05 16:34:03.120349 +0200][Error ][TlsMsg ] Failed to do TLS connect: Unable to connect to cmsio7.rc.ufl.edu; error_ssl > [2023-09-05 16:34:03.120356 +0200][Error ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] Socket error while handshaking: [FATAL] TLS error: resource temporarily unavailable > [2023-09-05 16:34:03.120361 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] Closing the socket > > The errno=104 means "Connection reset by peer". There is some > misunderstanding between the client and the server about TLS. > However, in the log there is a second attempt which works: > > [2023-09-05 16:34:03.476199 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending out kXR_login request, username: amadio, cgi: xrd.cc=ch&xrd.tz=1&xrd.appname=python3.11&xrd.info=&xrd.hostname=gentoo.cern.ch&xrd.rn=v5.6.1, dual-stack: true, private IPv4: false, private IPv6: false > [2023-09-05 16:34:03.476211 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange. > [2023-09-05 16:34:03.604269 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange. > [2023-09-05 16:34:03.604804 +0200][Info ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake done. > [2023-09-05 16:34:03.722116 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Logged in, session: 58ee1900338200002700000065581b00 > [2023-09-05 16:34:03.722144 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Authentication is required: &P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096: > [2023-09-05 16:34:03.722149 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending authentication data > sec_Client: protocol request for host cmsio7.rc.ufl.edu token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:�����' > sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0' > [2023-09-05 16:34:03.722201 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Trying to authenticate using gsi > [2023-09-05 16:34:03.964272 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending more authentication data for gsi > 230905 16:34:03 65677 cryptossl_X509::CertType: certificate has 10 extensions > [2023-09-05 16:34:04.879014 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Authenticated with gsi. > > Can you please change root:// to roots:// in your python script and try > again? I see the problem only when using root://. I will investigate how > I can fix this (I think I know where the problem comes from) and let you > know when a patch is ready for testing. > > Best regards, > -Guilherme > > On Tue, Sep 05, 2023 at 09:30:23AM -0400, Bockjoo Kim wrote: >> Hi Guilherme >> >> On 9/5/23 04:43, Guilherme Amadio wrote: >>> Dear Bokjoo, >>> >>> On Mon, Sep 04, 2023 at 06:18:32PM +0200, Bockjoo Kim wrote: >>>> Hi Guilherme, >>>> >>>> I am not sure how to pass XrdSecDEBUG=1 and XrdSecPROTOCOL=gsi to the >>>> python script. >>> These are just environment variables, you can set them before calling >>> your command, with env XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python test.py, >>> or by using export XrdSecDEBUG=1 and export XrdSecPROTOCOL=gsi before >>> running the command. >> It does not provide any extra debug info: >> >> XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python openASAMTestFile.py >> XRootD Client Versin 5.6.1 >> XRootD Server Version v5.5.5 >> >> >> Open Status OK False [FATAL] TLS error: resource temporarily >> unavailable None >> >> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root, >> flags=OpenFlags.READ, timeout=90) >> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily unavailable" >> >> If I add XRD_LOGLEVEL=dump, it does dump logs with >> >> XRD_LOGLEVEL=Dump XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python >> openASAMTestFile.py openASAMTestFile.txt 2>&1 >> >> (See https://bockjoo.web.cern.ch/openASAMTestFile.txt for the output) >> >>>> I tried to set them through os.environ, but the output is not verbose. >>> I think these variables must be set before loading the XRootD libraries, >>> otherwise they may be set only after the client has checked for them and >>> will have no effect. >>> >>>> Below are the xrdfs command line outputs, though. Both works. >>> Does this mean that you no longer have a problem? >> Yes, only in the command line, but not with the python open. >>> Can you please provide >>> the output of the xrdgsitest command? >> voms-proxy-info -strength >> 4096 >> which xrdgsitest > /dev/null 2>&1 ; echo $? >> 1 >> >> I don't have the command in Florida. I have the v5.5.5 command line >> xrootd client >> >> but the v5.6.1 python xrootd client on an RHEL8 machine where >> https://bockjoo.web.cern.ch/openASAMTestFile.py >> >> fails. >> >> At CERN, I can not run it because I failed to install XRootD and >> pyxrootd because of a cmake3 failure. >> >> Also at CERN, >> >> xrdgsitest passes (See https://bockjoo.web.cern.ch/xrdgsitest.txt), but >> >> xrdgsitest does not recognize the env X509_USER_PROXY. >> >> So, I had to copy it to where xrdgsitest wants: >> >> -bash-4.2$ xrdgsitest >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> + proxy file: /tmp/x509up_u5082 not found >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> -bash-4.2$ cp $X509_USER_PROXY /tmp/x509up_u5082 >> -bash-4.2$ xrdgsitest >> >> || >> --------------------------------------------------------------------------------- >> || Crypto functionality tests for GSI >> ---------------------------------------------- >> || >> --------------------------------------------------------------------------------- >> || Loading EEC >> ............................................................. PASSED >> ... >> >> ... >> >> >>> If you don't have all PASSED as >>> result, then please send us the output of xrdgsitest -v too. Below is >>> the output I get. Just make sure to use "voms-proxy-init -bits 4096" >>> before calling xrdgsitest, as there's a bug (which I already have a fix >>> for) that if a user proxy certificate is not already set up it will >>> crash. >>> >>> I hope this helps in debugging the issue. If you get a TLS handshake >>> failure, like described in issue 2078, then I'd appreciate it if you >>> could export XRD_LOGLEVEL=Dump, and XrdSecDEBUG=1, re-run the command >>> and send us the full output. Also, it would help a lot if you tell me >>> how you are setting up your client proxy certificate for authentication. >> So, I am wondering how openASAMTestFile.py passes in your (5.6.1 XRootD, >> pyxrootd) >> >> environment for a 5.5.5 server with the TLS configuration. >> >> Thanks, >> >> Bockjoo >> >>> Best regards, >>> -Guilherme >>> >>> gentoo xrootd $ voms-proxy-info >>> subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio/CN=1089083835 >>> issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio >>> identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio >>> type : RFC compliant proxy >>> strength : 4096 bits >>> path : /tmp/x509up_u75748 >>> timeleft : 11:13:49 >>> gentoo xrootd $ xrdgsitest >>> || --------------------------------------------------------------------------------- >>> || Crypto functionality tests for GSI ---------------------------------------------- >>> || --------------------------------------------------------------------------------- >>> || Loading EEC ............................................................. PASSED >>> || Loading User Proxy ...................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Recreate the proxy certificate -------------------------------------------------- >>> Enter PEM pass phrase: >>> || Recreating User Proxy ................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Load CA certificates ------------------------------------------------------------ >>> || Loading CA certificate .................................................. PASSED >>> || Loading CA certificate .................................................. PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing ParseFile --------------------------------------------------------------- >>> || Chain reorder: ......................................................... PASSED >>> || Chain verify: .......................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing ExportChain ------------------------------------------------------------- >>> || Attach to X509ExportChain ............................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing Chain Import ------------------------------------------------------------ >>> || Chain reorder: ......................................................... PASSED >>> || Chain verify: .......................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing GSI chain import and verification --------------------------------------- >>> || GSI chain verify: ...................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing GSI chain copy ---------------------------------------------------------- >>> || GSI chain verify: ...................................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing Cert verification ------------------------------------------------------- >>> || verify cert: EE signed by CA ............................................ PASSED >>> || verify cert: PX signed by EE ............................................ PASSED >>> || verify cert: PX not signed by CA ........................................ PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing request creation -------------------------------------------------------- >>> || Creating request ........................................................ PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing request signature ------------------------------------------------------- >>> || Check proxyCertInfo extension ........................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing export of signed proxy -------------------------------------------------- >>> || Saving signed proxy chain to file ....................................... PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing CRL identification ------------------------------------------------------ >>> || Check CRL distribution points extension OK .............................. PASSED >>> || --------------------------------------------------------------------------------- >>> || Testing CRL loading ------------------------------------------------------------- >>> --2023-09-05 10:38:15-- http://cafiles.cern.ch/cafiles/crl/CERN%20Root%20Certification%20Authority%202.crl >>> Resolving cafiles.cern.ch... 188.184.101.153 >>> Connecting to cafiles.cern.ch|188.184.101.153|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1097 (1.1K) [application/pkix-crl] >>> Saving to: ‘/tmp/5168735f.0.crltmp’ >>> >>> /tmp/5168735f.0.crltmp 100%[=================================================>] 1.07K --.-KB/s in 0s >>> >>> 2023-09-05 10:38:15 (327 MB/s) - ‘/tmp/5168735f.0.crltmp’ saved [1097/1097] >>> >>> || Loading CA1 crl ......................................................... PASSED >>> || CRL signature OK ........................................................ PASSED >>> || --------------------------------------------------------------------------------- >>> >>> >>>> Thanks >>>> >>>> Bockjoo >>>> >>>> [1] at CERN >>>> >>>> -bash-4.2$ rpm -qf $(which xrdfs) >>>> >>>> xrootd-client-5.6.1-1.el7.x86_64 >>>> -bash-4.2$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs >>>> xroots://cmsio2.rc.ufl.edu query config version >>>> sec_Client: protocol request for host cmsio2.rc.ufl.edu >>>> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:Ga�' >>>> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so >>>> Secgsi ------------------------------------------------------------------- >>>> Secgsi Mode: client >>>> Secgsi Debug: 1 >>>> Secgsi CA dir: /etc/grid-security/certificates/ >>>> Secgsi CA verification level: verifyss >>>> Secgsi CRL dir: /etc/grid-security/certificates/ >>>> Secgsi CRL extension: .r0 >>>> Secgsi CRL check level: try >>>> Secgsi CRL refresh time: 86400 >>>> Secgsi Certificate: /afs/cern.ch/user/b/bockjoo/.globus/usercert.pem >>>> Secgsi Key: /afs/cern.ch/user/b/bockjoo/.globus/userkey.pem >>>> Secgsi Proxy file: /afs/cern.ch/user/b/bockjoo/.proxy >>>> Secgsi Proxy validity: 12:00 >>>> Secgsi Proxy dep length: 0 >>>> Secgsi Proxy bits: 512 >>>> Secgsi Proxy sign option: 1 >>>> Secgsi Proxy delegation option: 0 >>>> Secgsi Pure Cert/Key authentication allowed >>>> Secgsi Allowed server names: [*/]<target host name>[/*] >>>> Secgsi Crypto modules: ssl >>>> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc >>>> Secgsi MDigests: sha256 >>>> Secgsi Trusting DNS for hostname checking >>>> Secgsi ------------------------------------------------------------------- >>>> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0' >>>> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions >>>> 230904 18:04:27 4963 secgsi_VerifyCA: Warning: CA certificate not >>>> self-signed and integrity not checked: assuming OK (ba240aa8.0) >>>> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions >>>> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 5 extensions >>>> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 12 extensions >>>> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 10 extensions >>>> v5.5.5 >>>> >>>> [2] Florida >>>> >>>> [bockjoo@cms site-packages]$ rpm -qf $(which xrdfs) >>>> xrootd-client-5.5.5-1.2.osg36.el8.x86_64 >>>> [bockjoo@cms site-packages]$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs >>>> xroots://cmsio2.rc.ufl.edu query config version >>>> sec_Client: protocol request for host cmsio2.rc.ufl.edu >>>> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:' >>>> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so >>>> Secgsi ------------------------------------------------------------------- >>>> Secgsi Mode: client >>>> Secgsi Debug: 1 >>>> Secgsi CA dir: /etc/grid-security/certificates/ >>>> Secgsi CA verification level: verifyss >>>> Secgsi CRL dir: /etc/grid-security/certificates/ >>>> Secgsi CRL extension: .r0 >>>> Secgsi CRL check level: try >>>> Secgsi CRL refresh time: 86400 >>>> Secgsi Certificate: /home/bockjoo/.globus/usercert.pem >>>> Secgsi Key: /home/bockjoo/.globus/userkey.pem >>>> Secgsi Proxy file: /home/bockjoo/.cmsuser.proxy >>>> Secgsi Proxy validity: 12:00 >>>> Secgsi Proxy dep length: 0 >>>> Secgsi Proxy bits: 512 >>>> Secgsi Proxy sign option: 1 >>>> Secgsi Proxy delegation option: 0 >>>> Secgsi Pure Cert/Key authentication allowed >>>> Secgsi Allowed server names: [*/]<target host name>[/*] >>>> Secgsi Crypto modules: ssl >>>> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc >>>> Secgsi MDigests: sha1:md5 >>>> Secgsi Trusting DNS for hostname checking >>>> Secgsi ------------------------------------------------------------------- >>>> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0' >>>> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8 >>>> extensions >>>> 230904 12:10:43 1107303 secgsi_VerifyCA: Warning: CA certificate not >>>> self-signed and integrity not checked: assuming OK (ba240aa8.0) >>>> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8 >>>> extensions >>>> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 5 >>>> extensions >>>> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 12 >>>> extensions >>>> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 10 >>>> extensions >>>> v5.5.5 >>>> >>>> On 9/4/23 09:53, Guilherme Amadio wrote: >>>>> Dear Bokjoo, >>>>> >>>>> I tried your script, and I can connect without problems to 5.5 servers >>>>> here at CERN where I have access via gsi protocol. I get permission denied >>>>> on your server, but no TLS error. I tried both from CentOS 7 (OpenSSL >>>>> 1.0.x) and from my own machine (where I use OpenSSL 1.1.x) with client v5.6.1. >>>>> >>>>> $ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs xroots://eoscms.cern.ch query config version >>>>> sec_Client: protocol request for host eoscms.cern.ch token='&P=krb5,[log in to unmask]&P=gsi,v:10600,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.+13:/etc/eos.keytab&P=unix' >>>>> sec_PM: Skipping krb5 only want gsi >>>>> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so >>>>> Secgsi ------------------------------------------------------------------- >>>>> Secgsi Mode: client >>>>> Secgsi Debug: 1 >>>>> Secgsi CA dir: /etc/grid-security/certificates/ >>>>> Secgsi CA verification level: verifyss >>>>> Secgsi CRL dir: /etc/grid-security/certificates/ >>>>> Secgsi CRL extension: .r0 >>>>> Secgsi CRL check level: try >>>>> Secgsi CRL refresh time: 86400 >>>>> Secgsi Certificate: /home/amadio/.globus/usercert.pem >>>>> Secgsi Key: /home/amadio/.globus/userkey.pem >>>>> Secgsi Proxy file: /tmp/x509up_u75748 >>>>> Secgsi Proxy validity: 12:00 >>>>> Secgsi Proxy dep length: 0 >>>>> Secgsi Proxy bits: 512 >>>>> Secgsi Proxy sign option: 1 >>>>> Secgsi Proxy delegation option: 0 >>>>> Secgsi Pure Cert/Key authentication allowed >>>>> Secgsi Allowed server names: [*/]<target host name>[/*] >>>>> Secgsi Crypto modules: ssl >>>>> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc >>>>> Secgsi MDigests: sha256 >>>>> Secgsi Trusting DNS for hostname checking >>>>> Secgsi ------------------------------------------------------------------- >>>>> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:5168735f.0|4339b4bc.0' >>>>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions >>>>> 230904 10:30:53 125169 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (5168735f.0) >>>>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions >>>>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 4 extensions >>>>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 12 extensions >>>>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions >>>>> 5.5.10 >>>>> >>>>> Could you please post a verbose output of the error you see, like I did >>>>> above for the working case? I think that the client might be missing >>>>> some configuration or some certificates which are required to validate >>>>> the server and that's why you see errors. With XrdSecDEBUG=1 XrdSecPROTOCOL=gsi >>>>> we should be able to get enough information to debug this. >>>>> >>>>> Best regards, >>>>> -Guilherme >>>>> >>>>> On Tue, Aug 29, 2023 at 12:39:40PM -0400, Bockjoo Kim wrote: >>>>>> Yes, as I posted in the original email: "If I switch to 5.5.5 client, there is no issue" >>>>>> Thanks, >>>>>> Bockjoo >>>>>> >>>>>> On 8/29/23 12:11, Brian Lin wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> We're currently seeing similar TLS issues in osg-test with xrdcp. It's >>>>>>> why we haven't released 5.6.x in the OSG repos yet: >>>>>>> https://opensciencegrid.atlassian.net/browse/SOFTWARE-5623?focusedCommentId=380033. >>>>>>> >>>>>>> I'm also CC'ing Lincoln who is seeing this issue out in the wild with >>>>>>> a 5.6.1 client against a 5.5.5 server (see attached). For comparison, >>>>>>> I've also attached a successful copy for a 5.5.5 client vs 5.5.5 >>>>>>> server for the same file. >>>>>>> >>>>>>> Bockjoo: would you be able to try downgrading to a 5.5 client and >>>>>>> seeing if that resolves your issue? >>>>>>> >>>>>>> Thanks, >>>>>>> Brian >>>>>>> >>>>>>> >>>>>>> On 8/28/23 08:42, Bockjoo Kim wrote: >>>>>>>> Hi Guilherme, >>>>>>>> >>>>>>>> The client machine (xrootd 5.6.1) that I had the issue is a fully >>>>>>>> credible OSG CE. >>>>>>>> >>>>>>>> So, there should be no issue with the CA and X509_CERT_DIR. >>>>>>>> >>>>>>>> I have used this python script to reproduce the issue on the client >>>>>>>> machine: >>>>>>>> >>>>>>>> ############################################# >>>>>>>> >>>>>>>> import os >>>>>>>> import sys >>>>>>>> import errno >>>>>>>> import subprocess >>>>>>>> import zlib >>>>>>>> import random >>>>>>>> from XRootD import client >>>>>>>> from XRootD.client.flags import OpenFlags >>>>>>>> >>>>>>>> ENDPOINT='cmsio2.rc.ufl.edu:1094' >>>>>>>> SAM_TEST_FILE='/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root' >>>>>>>> >>>>>>>> >>>>>>>> print ("XRootD Client Versin",client.__version__) >>>>>>>> cmd = [ "xrdfs "+ENDPOINT+" query config version" ] >>>>>>>> try: >>>>>>>> result = subprocess.run(cmd, shell=True, capture_output=True, >>>>>>>> text=True) >>>>>>>> print("XRootD Server Version", result.stdout) >>>>>>>> except subprocess.TimeoutExpired: >>>>>>>> print("connecting to endpoint timed out") >>>>>>>> >>>>>>>> >>>>>>>> os.environ["X509_CERT_DIR"] = >>>>>>>> "/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates" >>>>>>>> os.environ["X509_USER_PROXY"] = "/home/bockjoo/.cmsuser.proxy" >>>>>>>> os.environ["X509_USER_PROXY_NONCMS"] = "/home/bockjoo/.griduser.proxy" >>>>>>>> os.environ["XRD_NETWORKSTACK"] = "IPv4" >>>>>>>> with client.File() as f: >>>>>>>> status, response = f.open("root://" + ENDPOINT + "/" + \ >>>>>>>> SAM_TEST_FILE, flags=OpenFlags.READ, timeout=90) >>>>>>>> if ( not status.ok ): >>>>>>>> print (("\nopen(root://%s/%s, flags=OpenFlags.READ, >>>>>>>> time" + \ >>>>>>>> "out=90)\nXRootDStatus.code=%d \"%s\"\n") % \ >>>>>>>> (ENDPOINT, SAM_TEST_FILE, status.code, \ >>>>>>>> status.message.replace("\n", ""))) >>>>>>>> pass >>>>>>>> status, data = f.read(offset=0, size=65536, timeout=90) >>>>>>>> if ( not status.ok ): >>>>>>>> print(("\n%s\nread(offset=0, size=65536, >>>>>>>> timeout=90)\n" + \ >>>>>>>> "XRootDStatus.code=%d \"%s\"\n") % >>>>>>>> (SAM_TEST_FILE, \ >>>>>>>> status.code, status.message.replace("\n", >>>>>>>> ""))) >>>>>>>> pass >>>>>>>> print ("Open Status",status.ok) >>>>>>>> >>>>>>>> ############################################# >>>>>>>> >>>>>>>> You can choose the endpoint and the file of your choosing with the >>>>>>>> 5.5.5 server >>>>>>>> >>>>>>>> to test it. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Bockjoo >>>>>>>> >>>>>>>> On 8/28/23 09:33, Guilherme Amadio wrote: >>>>>>>>> Dear Bockjoo, >>>>>>>>> >>>>>>>>> On Sat, Aug 26, 2023 at 04:14:28PM -0400, Bockjoo Kim wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am seeing a python XRootD file open issue for the 5.6.1 client >>>>>>>>>> with a >>>>>>>>>> 5.5.5 server : >>>>>>>>>> >>>>>>>>>> ============================================================= >>>>>>>>>> >>>>>>>>>> XRootD Client Versin 5.6.1 >>>>>>>>>> XRootD Server Version v5.5.5 >>>>>>>>>> >>>>>>>>>> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root, >>>>>>>>>> >>>>>>>>>> flags=OpenFlags.READ, timeout=90) >>>>>>>>>> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily >>>>>>>>>> unavailable: Unable to connect to cmsio2.rc.ufl.edu; error_ssl" >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> ValueError Traceback (most recent >>>>>>>>>> call last) >>>>>>>>>> /tmp/ipykernel_4179061/812350213.py in <module> >>>>>>>>>> 40 status.message.replace("\n", ""))) >>>>>>>>>> 41 #pass >>>>>>>>>> ---> 42 status, data = f.read(offset=0, size=65536, >>>>>>>>>> timeout=90) >>>>>>>>>> 43 if ( not status.ok ): >>>>>>>>>> 44 print(("\n%s\nread(offset=0, size=65536, >>>>>>>>>> timeout=90)\n" + \ >>>>>>>>>> >>>>>>>>>> /opt/cms/services/anaconda3/lib/python3.9/site-packages/XRootD/client/file.py >>>>>>>>>> >>>>>>>>>> in read(self, offset, size, timeout, callback) >>>>>>>>>> 124 return XRootDStatus(self.__file.read(offset, size, >>>>>>>>>> timeout, callback)) >>>>>>>>>> 125 >>>>>>>>>> --> 126 status, response = self.__file.read(offset, size, timeout) >>>>>>>>>> 127 return XRootDStatus(status), response >>>>>>>>>> 128 >>>>>>>>>> >>>>>>>>>> ValueError: I/O operation on closed file >>>>>>>>>> >>>>>>>>>> =============================================================== >>>>>>>>>> >>>>>>>>>> Here, XRootD Server is configured with TLS. >>>>>>>>>> >>>>>>>>>> If I remove TLS configuration of the 5.5.5 server, there is no issue. >>>>>>>>>> >>>>>>>>>> If I switch to 5.5.5 client, there is no issue. >>>>>>>>>> >>>>>>>>>> Is this expected? >>>>>>>>> It may or may not be. When I wrote the patch, I tested several >>>>>>>>> scenarios >>>>>>>>> (see >>>>>>>>> https://github.com/xrootd/xrootd/pull/2031#issuecomment-1589380486). >>>>>>>>> The error message that you see is likely caused by a client that cannot >>>>>>>>> validate the server with TLS (because it does not have the proper CA >>>>>>>>> certificates installed locally). So I suggest you to try with xrdcp >>>>>>>>> --notlsok option, or export X509_CERT_DIR=/dev/null to force the client >>>>>>>>> into not being able to do TLS at all. If the directory >>>>>>>>> /etc/grid-security >>>>>>>>> exists on your machine, but the client cannot verify the server, and >>>>>>>>> TLS >>>>>>>>> is enforced, then this behavior is expected. Otherwise, please export >>>>>>>>> XRD_LOGLEVEL=Dump, re-run the command and send us the output so I can >>>>>>>>> investigate this issue further. You may also want to install the proper >>>>>>>>> certificates to let the client validate the server to be able to use >>>>>>>>> TLS. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> -Guilherme >>>>>>>> ######################################################################## >>>>>>>> Use REPLY-ALL to reply to list >>>>>>>> >>>>>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1