Dear Bokjoo,
I think I've finally managed to reproduce this problem. The output log I
get is attached. In it, I see:
[2023-09-05 16:34:03.004973 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending out kXR_login request, username: amadio, cgi: xrd.cc=ch&xrd.tz=1&xrd.appname=python3.11&xrd.info=&xrd.hostname=gentoo.cern.ch&xrd.rn=v5.6.1, dual-stack: true, private IPv4: false, private IPv6: false
2023-09-05 16:34:03.004985 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange.
[2023-09-05 16:34:03.120274 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange.
[2023-09-05 16:34:03.120329 +0200][Error ][TlsMsg ] [] TLS error rc=-1 ec=1 (error_ssl) errno=104.
[2023-09-05 16:34:03.120342 +0200][Debug ][TlsMsg ] [] 140201906132672:error:1408F0C6:SSL routines:ssl3_get_record:packet length too long:../openssl-1.1.1v/ssl/record/ssl3_record.c:362:
[2023-09-05 16:34:03.120349 +0200][Error ][TlsMsg ] Failed to do TLS connect: Unable to connect to cmsio7.rc.ufl.edu; error_ssl
[2023-09-05 16:34:03.120356 +0200][Error ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] Socket error while handshaking: [FATAL] TLS error: resource temporarily unavailable
[2023-09-05 16:34:03.120361 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] Closing the socket
The errno=104 means "Connection reset by peer". There is some
misunderstanding between the client and the server about TLS.
However, in the log there is a second attempt which works:
[2023-09-05 16:34:03.476199 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending out kXR_login request, username: amadio, cgi: xrd.cc=ch&xrd.tz=1&xrd.appname=python3.11&xrd.info=&xrd.hostname=gentoo.cern.ch&xrd.rn=v5.6.1, dual-stack: true, private IPv4: false, private IPv6: false
[2023-09-05 16:34:03.476211 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange.
[2023-09-05 16:34:03.604269 +0200][Debug ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake exchange.
[2023-09-05 16:34:03.604804 +0200][Info ][AsyncSock ] [cmsio7.rc.ufl.edu:1094.0] TLS hand-shake done.
[2023-09-05 16:34:03.722116 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Logged in, session: 58ee1900338200002700000065581b00
[2023-09-05 16:34:03.722144 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Authentication is required: &P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:
[2023-09-05 16:34:03.722149 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending authentication data
sec_Client: protocol request for host cmsio7.rc.ufl.edu token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:�����'
sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
[2023-09-05 16:34:03.722201 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Trying to authenticate using gsi
[2023-09-05 16:34:03.964272 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Sending more authentication data for gsi
230905 16:34:03 65677 cryptossl_X509::CertType: certificate has 10 extensions
[2023-09-05 16:34:04.879014 +0200][Debug ][XRootDTransport ] [cmsio7.rc.ufl.edu:1094.0] Authenticated with gsi.
Can you please change root:// to roots:// in your python script and try
again? I see the problem only when using root://. I will investigate how
I can fix this (I think I know where the problem comes from) and let you
know when a patch is ready for testing.
Best regards,
-Guilherme
On Tue, Sep 05, 2023 at 09:30:23AM -0400, Bockjoo Kim wrote:
> Hi Guilherme
>
> On 9/5/23 04:43, Guilherme Amadio wrote:
> > Dear Bokjoo,
> >
> > On Mon, Sep 04, 2023 at 06:18:32PM +0200, Bockjoo Kim wrote:
> >> Hi Guilherme,
> >>
> >> I am not sure how to pass XrdSecDEBUG=1 and XrdSecPROTOCOL=gsi to the
> >> python script.
> > These are just environment variables, you can set them before calling
> > your command, with env XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python test.py,
> > or by using export XrdSecDEBUG=1 and export XrdSecPROTOCOL=gsi before
> > running the command.
>
> It does not provide any extra debug info:
>
> XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python openASAMTestFile.py
> XRootD Client Versin 5.6.1
> XRootD Server Version v5.5.5
>
>
> Open Status OK False [FATAL] TLS error: resource temporarily
> unavailable None
>
> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root,
> flags=OpenFlags.READ, timeout=90)
> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily unavailable"
>
> If I add XRD_LOGLEVEL=dump, it does dump logs with
>
> XRD_LOGLEVEL=Dump XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python
> openASAMTestFile.py openASAMTestFile.txt 2>&1
>
> (See https://bockjoo.web.cern.ch/openASAMTestFile.txt for the output)
>
> >> I tried to set them through os.environ, but the output is not verbose.
> > I think these variables must be set before loading the XRootD libraries,
> > otherwise they may be set only after the client has checked for them and
> > will have no effect.
> >
> >> Below are the xrdfs command line outputs, though. Both works.
> > Does this mean that you no longer have a problem?
> Yes, only in the command line, but not with the python open.
> > Can you please provide
> > the output of the xrdgsitest command?
> voms-proxy-info -strength
> 4096
> which xrdgsitest > /dev/null 2>&1 ; echo $?
> 1
>
> I don't have the command in Florida. I have the v5.5.5 command line
> xrootd client
>
> but the v5.6.1 python xrootd client on an RHEL8 machine where
> https://bockjoo.web.cern.ch/openASAMTestFile.py
>
> fails.
>
> At CERN, I can not run it because I failed to install XRootD and
> pyxrootd because of a cmake3 failure.
>
> Also at CERN,
>
> xrdgsitest passes (See https://bockjoo.web.cern.ch/xrdgsitest.txt), but
>
> xrdgsitest does not recognize the env X509_USER_PROXY.
>
> So, I had to copy it to where xrdgsitest wants:
>
> -bash-4.2$ xrdgsitest
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> + proxy file: /tmp/x509up_u5082 not found
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> -bash-4.2$ cp $X509_USER_PROXY /tmp/x509up_u5082
> -bash-4.2$ xrdgsitest
>
> ||
> ---------------------------------------------------------------------------------
> || Crypto functionality tests for GSI
> ----------------------------------------------
> ||
> ---------------------------------------------------------------------------------
> || Loading EEC
> ............................................................. PASSED
> ...
>
> ...
>
>
> > If you don't have all PASSED as
> > result, then please send us the output of xrdgsitest -v too. Below is
> > the output I get. Just make sure to use "voms-proxy-init -bits 4096"
> > before calling xrdgsitest, as there's a bug (which I already have a fix
> > for) that if a user proxy certificate is not already set up it will
> > crash.
> >
> > I hope this helps in debugging the issue. If you get a TLS handshake
> > failure, like described in issue 2078, then I'd appreciate it if you
> > could export XRD_LOGLEVEL=Dump, and XrdSecDEBUG=1, re-run the command
> > and send us the full output. Also, it would help a lot if you tell me
> > how you are setting up your client proxy certificate for authentication.
>
> So, I am wondering how openASAMTestFile.py passes in your (5.6.1 XRootD,
> pyxrootd)
>
> environment for a 5.5.5 server with the TLS configuration.
>
> Thanks,
>
> Bockjoo
>
> > Best regards,
> > -Guilherme
> >
> > gentoo xrootd $ voms-proxy-info
> > subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio/CN=1089083835
> > issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio
> > identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio
> > type : RFC compliant proxy
> > strength : 4096 bits
> > path : /tmp/x509up_u75748
> > timeleft : 11:13:49
> > gentoo xrootd $ xrdgsitest
> > || ---------------------------------------------------------------------------------
> > || Crypto functionality tests for GSI ----------------------------------------------
> > || ---------------------------------------------------------------------------------
> > || Loading EEC ............................................................. PASSED
> > || Loading User Proxy ...................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Recreate the proxy certificate --------------------------------------------------
> > Enter PEM pass phrase:
> > || Recreating User Proxy ................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Load CA certificates ------------------------------------------------------------
> > || Loading CA certificate .................................................. PASSED
> > || Loading CA certificate .................................................. PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing ParseFile ---------------------------------------------------------------
> > || Chain reorder: ......................................................... PASSED
> > || Chain verify: .......................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing ExportChain -------------------------------------------------------------
> > || Attach to X509ExportChain ............................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing Chain Import ------------------------------------------------------------
> > || Chain reorder: ......................................................... PASSED
> > || Chain verify: .......................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing GSI chain import and verification ---------------------------------------
> > || GSI chain verify: ...................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing GSI chain copy ----------------------------------------------------------
> > || GSI chain verify: ...................................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing Cert verification -------------------------------------------------------
> > || verify cert: EE signed by CA ............................................ PASSED
> > || verify cert: PX signed by EE ............................................ PASSED
> > || verify cert: PX not signed by CA ........................................ PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing request creation --------------------------------------------------------
> > || Creating request ........................................................ PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing request signature -------------------------------------------------------
> > || Check proxyCertInfo extension ........................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing export of signed proxy --------------------------------------------------
> > || Saving signed proxy chain to file ....................................... PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing CRL identification ------------------------------------------------------
> > || Check CRL distribution points extension OK .............................. PASSED
> > || ---------------------------------------------------------------------------------
> > || Testing CRL loading -------------------------------------------------------------
> > --2023-09-05 10:38:15-- http://cafiles.cern.ch/cafiles/crl/CERN%20Root%20Certification%20Authority%202.crl
> > Resolving cafiles.cern.ch... 188.184.101.153
> > Connecting to cafiles.cern.ch|188.184.101.153|:80... connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 1097 (1.1K) [application/pkix-crl]
> > Saving to: ‘/tmp/5168735f.0.crltmp’
> >
> > /tmp/5168735f.0.crltmp 100%[=================================================>] 1.07K --.-KB/s in 0s
> >
> > 2023-09-05 10:38:15 (327 MB/s) - ‘/tmp/5168735f.0.crltmp’ saved [1097/1097]
> >
> > || Loading CA1 crl ......................................................... PASSED
> > || CRL signature OK ........................................................ PASSED
> > || ---------------------------------------------------------------------------------
> >
> >
> >> Thanks
> >>
> >> Bockjoo
> >>
> >> [1] at CERN
> >>
> >> -bash-4.2$ rpm -qf $(which xrdfs)
> >>
> >> xrootd-client-5.6.1-1.el7.x86_64
> >> -bash-4.2$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
> >> xroots://cmsio2.rc.ufl.edu query config version
> >> sec_Client: protocol request for host cmsio2.rc.ufl.edu
> >> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:Ga�'
> >> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> >> Secgsi -------------------------------------------------------------------
> >> Secgsi Mode: client
> >> Secgsi Debug: 1
> >> Secgsi CA dir: /etc/grid-security/certificates/
> >> Secgsi CA verification level: verifyss
> >> Secgsi CRL dir: /etc/grid-security/certificates/
> >> Secgsi CRL extension: .r0
> >> Secgsi CRL check level: try
> >> Secgsi CRL refresh time: 86400
> >> Secgsi Certificate: /afs/cern.ch/user/b/bockjoo/.globus/usercert.pem
> >> Secgsi Key: /afs/cern.ch/user/b/bockjoo/.globus/userkey.pem
> >> Secgsi Proxy file: /afs/cern.ch/user/b/bockjoo/.proxy
> >> Secgsi Proxy validity: 12:00
> >> Secgsi Proxy dep length: 0
> >> Secgsi Proxy bits: 512
> >> Secgsi Proxy sign option: 1
> >> Secgsi Proxy delegation option: 0
> >> Secgsi Pure Cert/Key authentication allowed
> >> Secgsi Allowed server names: [*/]<target host name>[/*]
> >> Secgsi Crypto modules: ssl
> >> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> >> Secgsi MDigests: sha256
> >> Secgsi Trusting DNS for hostname checking
> >> Secgsi -------------------------------------------------------------------
> >> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
> >> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
> >> 230904 18:04:27 4963 secgsi_VerifyCA: Warning: CA certificate not
> >> self-signed and integrity not checked: assuming OK (ba240aa8.0)
> >> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
> >> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 5 extensions
> >> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 12 extensions
> >> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 10 extensions
> >> v5.5.5
> >>
> >> [2] Florida
> >>
> >> [bockjoo@cms site-packages]$ rpm -qf $(which xrdfs)
> >> xrootd-client-5.5.5-1.2.osg36.el8.x86_64
> >> [bockjoo@cms site-packages]$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
> >> xroots://cmsio2.rc.ufl.edu query config version
> >> sec_Client: protocol request for host cmsio2.rc.ufl.edu
> >> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:'
> >> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> >> Secgsi -------------------------------------------------------------------
> >> Secgsi Mode: client
> >> Secgsi Debug: 1
> >> Secgsi CA dir: /etc/grid-security/certificates/
> >> Secgsi CA verification level: verifyss
> >> Secgsi CRL dir: /etc/grid-security/certificates/
> >> Secgsi CRL extension: .r0
> >> Secgsi CRL check level: try
> >> Secgsi CRL refresh time: 86400
> >> Secgsi Certificate: /home/bockjoo/.globus/usercert.pem
> >> Secgsi Key: /home/bockjoo/.globus/userkey.pem
> >> Secgsi Proxy file: /home/bockjoo/.cmsuser.proxy
> >> Secgsi Proxy validity: 12:00
> >> Secgsi Proxy dep length: 0
> >> Secgsi Proxy bits: 512
> >> Secgsi Proxy sign option: 1
> >> Secgsi Proxy delegation option: 0
> >> Secgsi Pure Cert/Key authentication allowed
> >> Secgsi Allowed server names: [*/]<target host name>[/*]
> >> Secgsi Crypto modules: ssl
> >> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> >> Secgsi MDigests: sha1:md5
> >> Secgsi Trusting DNS for hostname checking
> >> Secgsi -------------------------------------------------------------------
> >> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
> >> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
> >> extensions
> >> 230904 12:10:43 1107303 secgsi_VerifyCA: Warning: CA certificate not
> >> self-signed and integrity not checked: assuming OK (ba240aa8.0)
> >> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
> >> extensions
> >> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 5
> >> extensions
> >> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 12
> >> extensions
> >> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 10
> >> extensions
> >> v5.5.5
> >>
> >> On 9/4/23 09:53, Guilherme Amadio wrote:
> >>> Dear Bokjoo,
> >>>
> >>> I tried your script, and I can connect without problems to 5.5 servers
> >>> here at CERN where I have access via gsi protocol. I get permission denied
> >>> on your server, but no TLS error. I tried both from CentOS 7 (OpenSSL
> >>> 1.0.x) and from my own machine (where I use OpenSSL 1.1.x) with client v5.6.1.
> >>>
> >>> $ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs xroots://eoscms.cern.ch query config version
> >>> sec_Client: protocol request for host eoscms.cern.ch token='&P=krb5,[log in to unmask]&P=gsi,v:10600,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.+13:/etc/eos.keytab&P=unix'
> >>> sec_PM: Skipping krb5 only want gsi
> >>> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> >>> Secgsi -------------------------------------------------------------------
> >>> Secgsi Mode: client
> >>> Secgsi Debug: 1
> >>> Secgsi CA dir: /etc/grid-security/certificates/
> >>> Secgsi CA verification level: verifyss
> >>> Secgsi CRL dir: /etc/grid-security/certificates/
> >>> Secgsi CRL extension: .r0
> >>> Secgsi CRL check level: try
> >>> Secgsi CRL refresh time: 86400
> >>> Secgsi Certificate: /home/amadio/.globus/usercert.pem
> >>> Secgsi Key: /home/amadio/.globus/userkey.pem
> >>> Secgsi Proxy file: /tmp/x509up_u75748
> >>> Secgsi Proxy validity: 12:00
> >>> Secgsi Proxy dep length: 0
> >>> Secgsi Proxy bits: 512
> >>> Secgsi Proxy sign option: 1
> >>> Secgsi Proxy delegation option: 0
> >>> Secgsi Pure Cert/Key authentication allowed
> >>> Secgsi Allowed server names: [*/]<target host name>[/*]
> >>> Secgsi Crypto modules: ssl
> >>> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> >>> Secgsi MDigests: sha256
> >>> Secgsi Trusting DNS for hostname checking
> >>> Secgsi -------------------------------------------------------------------
> >>> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:5168735f.0|4339b4bc.0'
> >>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> >>> 230904 10:30:53 125169 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (5168735f.0)
> >>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> >>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 4 extensions
> >>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 12 extensions
> >>> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> >>> 5.5.10
> >>>
> >>> Could you please post a verbose output of the error you see, like I did
> >>> above for the working case? I think that the client might be missing
> >>> some configuration or some certificates which are required to validate
> >>> the server and that's why you see errors. With XrdSecDEBUG=1 XrdSecPROTOCOL=gsi
> >>> we should be able to get enough information to debug this.
> >>>
> >>> Best regards,
> >>> -Guilherme
> >>>
> >>> On Tue, Aug 29, 2023 at 12:39:40PM -0400, Bockjoo Kim wrote:
> >>>> Yes, as I posted in the original email: "If I switch to 5.5.5 client, there is no issue"
> >>>> Thanks,
> >>>> Bockjoo
> >>>>
> >>>> On 8/29/23 12:11, Brian Lin wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> We're currently seeing similar TLS issues in osg-test with xrdcp. It's
> >>>>> why we haven't released 5.6.x in the OSG repos yet:
> >>>>> https://opensciencegrid.atlassian.net/browse/SOFTWARE-5623?focusedCommentId=380033.
> >>>>>
> >>>>> I'm also CC'ing Lincoln who is seeing this issue out in the wild with
> >>>>> a 5.6.1 client against a 5.5.5 server (see attached). For comparison,
> >>>>> I've also attached a successful copy for a 5.5.5 client vs 5.5.5
> >>>>> server for the same file.
> >>>>>
> >>>>> Bockjoo: would you be able to try downgrading to a 5.5 client and
> >>>>> seeing if that resolves your issue?
> >>>>>
> >>>>> Thanks,
> >>>>> Brian
> >>>>>
> >>>>>
> >>>>> On 8/28/23 08:42, Bockjoo Kim wrote:
> >>>>>> Hi Guilherme,
> >>>>>>
> >>>>>> The client machine (xrootd 5.6.1) that I had the issue is a fully
> >>>>>> credible OSG CE.
> >>>>>>
> >>>>>> So, there should be no issue with the CA and X509_CERT_DIR.
> >>>>>>
> >>>>>> I have used this python script to reproduce the issue on the client
> >>>>>> machine:
> >>>>>>
> >>>>>> #############################################
> >>>>>>
> >>>>>> import os
> >>>>>> import sys
> >>>>>> import errno
> >>>>>> import subprocess
> >>>>>> import zlib
> >>>>>> import random
> >>>>>> from XRootD import client
> >>>>>> from XRootD.client.flags import OpenFlags
> >>>>>>
> >>>>>> ENDPOINT='cmsio2.rc.ufl.edu:1094'
> >>>>>> SAM_TEST_FILE='/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root'
> >>>>>>
> >>>>>>
> >>>>>> print ("XRootD Client Versin",client.__version__)
> >>>>>> cmd = [ "xrdfs "+ENDPOINT+" query config version" ]
> >>>>>> try:
> >>>>>> result = subprocess.run(cmd, shell=True, capture_output=True,
> >>>>>> text=True)
> >>>>>> print("XRootD Server Version", result.stdout)
> >>>>>> except subprocess.TimeoutExpired:
> >>>>>> print("connecting to endpoint timed out")
> >>>>>>
> >>>>>>
> >>>>>> os.environ["X509_CERT_DIR"] =
> >>>>>> "/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates"
> >>>>>> os.environ["X509_USER_PROXY"] = "/home/bockjoo/.cmsuser.proxy"
> >>>>>> os.environ["X509_USER_PROXY_NONCMS"] = "/home/bockjoo/.griduser.proxy"
> >>>>>> os.environ["XRD_NETWORKSTACK"] = "IPv4"
> >>>>>> with client.File() as f:
> >>>>>> status, response = f.open("root://" + ENDPOINT + "/" + \
> >>>>>> SAM_TEST_FILE, flags=OpenFlags.READ, timeout=90)
> >>>>>> if ( not status.ok ):
> >>>>>> print (("\nopen(root://%s/%s, flags=OpenFlags.READ,
> >>>>>> time" + \
> >>>>>> "out=90)\nXRootDStatus.code=%d \"%s\"\n") % \
> >>>>>> (ENDPOINT, SAM_TEST_FILE, status.code, \
> >>>>>> status.message.replace("\n", "")))
> >>>>>> pass
> >>>>>> status, data = f.read(offset=0, size=65536, timeout=90)
> >>>>>> if ( not status.ok ):
> >>>>>> print(("\n%s\nread(offset=0, size=65536,
> >>>>>> timeout=90)\n" + \
> >>>>>> "XRootDStatus.code=%d \"%s\"\n") %
> >>>>>> (SAM_TEST_FILE, \
> >>>>>> status.code, status.message.replace("\n",
> >>>>>> "")))
> >>>>>> pass
> >>>>>> print ("Open Status",status.ok)
> >>>>>>
> >>>>>> #############################################
> >>>>>>
> >>>>>> You can choose the endpoint and the file of your choosing with the
> >>>>>> 5.5.5 server
> >>>>>>
> >>>>>> to test it.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Bockjoo
> >>>>>>
> >>>>>> On 8/28/23 09:33, Guilherme Amadio wrote:
> >>>>>>> Dear Bockjoo,
> >>>>>>>
> >>>>>>> On Sat, Aug 26, 2023 at 04:14:28PM -0400, Bockjoo Kim wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I am seeing a python XRootD file open issue for the 5.6.1 client
> >>>>>>>> with a
> >>>>>>>> 5.5.5 server :
> >>>>>>>>
> >>>>>>>> =============================================================
> >>>>>>>>
> >>>>>>>> XRootD Client Versin 5.6.1
> >>>>>>>> XRootD Server Version v5.5.5
> >>>>>>>>
> >>>>>>>> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root,
> >>>>>>>>
> >>>>>>>> flags=OpenFlags.READ, timeout=90)
> >>>>>>>> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily
> >>>>>>>> unavailable: Unable to connect to cmsio2.rc.ufl.edu; error_ssl"
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------------
> >>>>>>>>
> >>>>>>>> ValueError Traceback (most recent
> >>>>>>>> call last)
> >>>>>>>> /tmp/ipykernel_4179061/812350213.py in <module>
> >>>>>>>> 40 status.message.replace("\n", "")))
> >>>>>>>> 41 #pass
> >>>>>>>> ---> 42 status, data = f.read(offset=0, size=65536,
> >>>>>>>> timeout=90)
> >>>>>>>> 43 if ( not status.ok ):
> >>>>>>>> 44 print(("\n%s\nread(offset=0, size=65536,
> >>>>>>>> timeout=90)\n" + \
> >>>>>>>>
> >>>>>>>> /opt/cms/services/anaconda3/lib/python3.9/site-packages/XRootD/client/file.py
> >>>>>>>>
> >>>>>>>> in read(self, offset, size, timeout, callback)
> >>>>>>>> 124 return XRootDStatus(self.__file.read(offset, size,
> >>>>>>>> timeout, callback))
> >>>>>>>> 125
> >>>>>>>> --> 126 status, response = self.__file.read(offset, size, timeout)
> >>>>>>>> 127 return XRootDStatus(status), response
> >>>>>>>> 128
> >>>>>>>>
> >>>>>>>> ValueError: I/O operation on closed file
> >>>>>>>>
> >>>>>>>> ===============================================================
> >>>>>>>>
> >>>>>>>> Here, XRootD Server is configured with TLS.
> >>>>>>>>
> >>>>>>>> If I remove TLS configuration of the 5.5.5 server, there is no issue.
> >>>>>>>>
> >>>>>>>> If I switch to 5.5.5 client, there is no issue.
> >>>>>>>>
> >>>>>>>> Is this expected?
> >>>>>>> It may or may not be. When I wrote the patch, I tested several
> >>>>>>> scenarios
> >>>>>>> (see
> >>>>>>> https://github.com/xrootd/xrootd/pull/2031#issuecomment-1589380486).
> >>>>>>> The error message that you see is likely caused by a client that cannot
> >>>>>>> validate the server with TLS (because it does not have the proper CA
> >>>>>>> certificates installed locally). So I suggest you to try with xrdcp
> >>>>>>> --notlsok option, or export X509_CERT_DIR=/dev/null to force the client
> >>>>>>> into not being able to do TLS at all. If the directory
> >>>>>>> /etc/grid-security
> >>>>>>> exists on your machine, but the client cannot verify the server, and
> >>>>>>> TLS
> >>>>>>> is enforced, then this behavior is expected. Otherwise, please export
> >>>>>>> XRD_LOGLEVEL=Dump, re-run the command and send us the output so I can
> >>>>>>> investigate this issue further. You may also want to install the proper
> >>>>>>> certificates to let the client validate the server to be able to use
> >>>>>>> TLS.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> -Guilherme
> >>>>>> ########################################################################
> >>>>>> Use REPLY-ALL to reply to list
> >>>>>>
> >>>>>> To unsubscribe from the XROOTD-L list, click the following link:
> >>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
