Dear Bokjoo,
On Mon, Sep 04, 2023 at 06:18:32PM +0200, Bockjoo Kim wrote:
> Hi Guilherme,
>
> I am not sure how to pass XrdSecDEBUG=1 and XrdSecPROTOCOL=gsi to the
> python script.
These are just environment variables, you can set them before calling
your command, with env XrdSecDEBUG=1 XrdSecPROTOCOL=gsi python test.py,
or by using export XrdSecDEBUG=1 and export XrdSecPROTOCOL=gsi before
running the command.
> I tried to set them through os.environ, but the output is not verbose.
I think these variables must be set before loading the XRootD libraries,
otherwise they may be set only after the client has checked for them and
will have no effect.
> Below are the xrdfs command line outputs, though. Both works.
Does this mean that you no longer have a problem? Can you please provide
the output of the xrdgsitest command? If you don't have all PASSED as
result, then please send us the output of xrdgsitest -v too. Below is
the output I get. Just make sure to use "voms-proxy-init -bits 4096"
before calling xrdgsitest, as there's a bug (which I already have a fix
for) that if a user proxy certificate is not already set up it will
crash.
I hope this helps in debugging the issue. If you get a TLS handshake
failure, like described in issue 2078, then I'd appreciate it if you
could export XRD_LOGLEVEL=Dump, and XrdSecDEBUG=1, re-run the command
and send us the full output. Also, it would help a lot if you tell me
how you are setting up your client proxy certificate for authentication.
Best regards,
-Guilherme
gentoo xrootd $ voms-proxy-info
subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio/CN=1089083835
issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio
identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amadio/CN=764132/CN=Guilherme Amadio
type : RFC compliant proxy
strength : 4096 bits
path : /tmp/x509up_u75748
timeleft : 11:13:49
gentoo xrootd $ xrdgsitest
|| ---------------------------------------------------------------------------------
|| Crypto functionality tests for GSI ----------------------------------------------
|| ---------------------------------------------------------------------------------
|| Loading EEC ............................................................. PASSED
|| Loading User Proxy ...................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Recreate the proxy certificate --------------------------------------------------
Enter PEM pass phrase:
|| Recreating User Proxy ................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Load CA certificates ------------------------------------------------------------
|| Loading CA certificate .................................................. PASSED
|| Loading CA certificate .................................................. PASSED
|| ---------------------------------------------------------------------------------
|| Testing ParseFile ---------------------------------------------------------------
|| Chain reorder: ......................................................... PASSED
|| Chain verify: .......................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing ExportChain -------------------------------------------------------------
|| Attach to X509ExportChain ............................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing Chain Import ------------------------------------------------------------
|| Chain reorder: ......................................................... PASSED
|| Chain verify: .......................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing GSI chain import and verification ---------------------------------------
|| GSI chain verify: ...................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing GSI chain copy ----------------------------------------------------------
|| GSI chain verify: ...................................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing Cert verification -------------------------------------------------------
|| verify cert: EE signed by CA ............................................ PASSED
|| verify cert: PX signed by EE ............................................ PASSED
|| verify cert: PX not signed by CA ........................................ PASSED
|| ---------------------------------------------------------------------------------
|| Testing request creation --------------------------------------------------------
|| Creating request ........................................................ PASSED
|| ---------------------------------------------------------------------------------
|| Testing request signature -------------------------------------------------------
|| Check proxyCertInfo extension ........................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing export of signed proxy --------------------------------------------------
|| Saving signed proxy chain to file ....................................... PASSED
|| ---------------------------------------------------------------------------------
|| Testing CRL identification ------------------------------------------------------
|| Check CRL distribution points extension OK .............................. PASSED
|| ---------------------------------------------------------------------------------
|| Testing CRL loading -------------------------------------------------------------
--2023-09-05 10:38:15-- http://cafiles.cern.ch/cafiles/crl/CERN%20Root%20Certification%20Authority%202.crl
Resolving cafiles.cern.ch... 188.184.101.153
Connecting to cafiles.cern.ch|188.184.101.153|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1097 (1.1K) [application/pkix-crl]
Saving to: ‘/tmp/5168735f.0.crltmp’
/tmp/5168735f.0.crltmp 100%[=================================================>] 1.07K --.-KB/s in 0s
2023-09-05 10:38:15 (327 MB/s) - ‘/tmp/5168735f.0.crltmp’ saved [1097/1097]
|| Loading CA1 crl ......................................................... PASSED
|| CRL signature OK ........................................................ PASSED
|| ---------------------------------------------------------------------------------
>
> Thanks
>
> Bockjoo
>
> [1] at CERN
>
> -bash-4.2$ rpm -qf $(which xrdfs)
>
> xrootd-client-5.6.1-1.el7.x86_64
> -bash-4.2$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
> xroots://cmsio2.rc.ufl.edu query config version
> sec_Client: protocol request for host cmsio2.rc.ufl.edu
> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:Ga�'
> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> Secgsi -------------------------------------------------------------------
> Secgsi Mode: client
> Secgsi Debug: 1
> Secgsi CA dir: /etc/grid-security/certificates/
> Secgsi CA verification level: verifyss
> Secgsi CRL dir: /etc/grid-security/certificates/
> Secgsi CRL extension: .r0
> Secgsi CRL check level: try
> Secgsi CRL refresh time: 86400
> Secgsi Certificate: /afs/cern.ch/user/b/bockjoo/.globus/usercert.pem
> Secgsi Key: /afs/cern.ch/user/b/bockjoo/.globus/userkey.pem
> Secgsi Proxy file: /afs/cern.ch/user/b/bockjoo/.proxy
> Secgsi Proxy validity: 12:00
> Secgsi Proxy dep length: 0
> Secgsi Proxy bits: 512
> Secgsi Proxy sign option: 1
> Secgsi Proxy delegation option: 0
> Secgsi Pure Cert/Key authentication allowed
> Secgsi Allowed server names: [*/]<target host name>[/*]
> Secgsi Crypto modules: ssl
> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> Secgsi MDigests: sha256
> Secgsi Trusting DNS for hostname checking
> Secgsi -------------------------------------------------------------------
> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
> 230904 18:04:27 4963 secgsi_VerifyCA: Warning: CA certificate not
> self-signed and integrity not checked: assuming OK (ba240aa8.0)
> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 5 extensions
> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 12 extensions
> 230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 10 extensions
> v5.5.5
>
> [2] Florida
>
> [bockjoo@cms site-packages]$ rpm -qf $(which xrdfs)
> xrootd-client-5.5.5-1.2.osg36.el8.x86_64
> [bockjoo@cms site-packages]$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
> xroots://cmsio2.rc.ufl.edu query config version
> sec_Client: protocol request for host cmsio2.rc.ufl.edu
> token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:'
> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> Secgsi -------------------------------------------------------------------
> Secgsi Mode: client
> Secgsi Debug: 1
> Secgsi CA dir: /etc/grid-security/certificates/
> Secgsi CA verification level: verifyss
> Secgsi CRL dir: /etc/grid-security/certificates/
> Secgsi CRL extension: .r0
> Secgsi CRL check level: try
> Secgsi CRL refresh time: 86400
> Secgsi Certificate: /home/bockjoo/.globus/usercert.pem
> Secgsi Key: /home/bockjoo/.globus/userkey.pem
> Secgsi Proxy file: /home/bockjoo/.cmsuser.proxy
> Secgsi Proxy validity: 12:00
> Secgsi Proxy dep length: 0
> Secgsi Proxy bits: 512
> Secgsi Proxy sign option: 1
> Secgsi Proxy delegation option: 0
> Secgsi Pure Cert/Key authentication allowed
> Secgsi Allowed server names: [*/]<target host name>[/*]
> Secgsi Crypto modules: ssl
> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> Secgsi MDigests: sha1:md5
> Secgsi Trusting DNS for hostname checking
> Secgsi -------------------------------------------------------------------
> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
> extensions
> 230904 12:10:43 1107303 secgsi_VerifyCA: Warning: CA certificate not
> self-signed and integrity not checked: assuming OK (ba240aa8.0)
> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
> extensions
> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 5
> extensions
> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 12
> extensions
> 230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 10
> extensions
> v5.5.5
>
> On 9/4/23 09:53, Guilherme Amadio wrote:
> > Dear Bokjoo,
> >
> > I tried your script, and I can connect without problems to 5.5 servers
> > here at CERN where I have access via gsi protocol. I get permission denied
> > on your server, but no TLS error. I tried both from CentOS 7 (OpenSSL
> > 1.0.x) and from my own machine (where I use OpenSSL 1.1.x) with client v5.6.1.
> >
> > $ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs xroots://eoscms.cern.ch query config version
> > sec_Client: protocol request for host eoscms.cern.ch token='&P=krb5,[log in to unmask]&P=gsi,v:10600,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.+13:/etc/eos.keytab&P=unix'
> > sec_PM: Skipping krb5 only want gsi
> > sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> > Secgsi -------------------------------------------------------------------
> > Secgsi Mode: client
> > Secgsi Debug: 1
> > Secgsi CA dir: /etc/grid-security/certificates/
> > Secgsi CA verification level: verifyss
> > Secgsi CRL dir: /etc/grid-security/certificates/
> > Secgsi CRL extension: .r0
> > Secgsi CRL check level: try
> > Secgsi CRL refresh time: 86400
> > Secgsi Certificate: /home/amadio/.globus/usercert.pem
> > Secgsi Key: /home/amadio/.globus/userkey.pem
> > Secgsi Proxy file: /tmp/x509up_u75748
> > Secgsi Proxy validity: 12:00
> > Secgsi Proxy dep length: 0
> > Secgsi Proxy bits: 512
> > Secgsi Proxy sign option: 1
> > Secgsi Proxy delegation option: 0
> > Secgsi Pure Cert/Key authentication allowed
> > Secgsi Allowed server names: [*/]<target host name>[/*]
> > Secgsi Crypto modules: ssl
> > Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> > Secgsi MDigests: sha256
> > Secgsi Trusting DNS for hostname checking
> > Secgsi -------------------------------------------------------------------
> > sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:5168735f.0|4339b4bc.0'
> > 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> > 230904 10:30:53 125169 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (5168735f.0)
> > 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> > 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 4 extensions
> > 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 12 extensions
> > 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> > 5.5.10
> >
> > Could you please post a verbose output of the error you see, like I did
> > above for the working case? I think that the client might be missing
> > some configuration or some certificates which are required to validate
> > the server and that's why you see errors. With XrdSecDEBUG=1 XrdSecPROTOCOL=gsi
> > we should be able to get enough information to debug this.
> >
> > Best regards,
> > -Guilherme
> >
> > On Tue, Aug 29, 2023 at 12:39:40PM -0400, Bockjoo Kim wrote:
> >> Yes, as I posted in the original email: "If I switch to 5.5.5 client, there is no issue"
> >> Thanks,
> >> Bockjoo
> >>
> >> On 8/29/23 12:11, Brian Lin wrote:
> >>> Hi all,
> >>>
> >>> We're currently seeing similar TLS issues in osg-test with xrdcp. It's
> >>> why we haven't released 5.6.x in the OSG repos yet:
> >>> https://opensciencegrid.atlassian.net/browse/SOFTWARE-5623?focusedCommentId=380033.
> >>>
> >>> I'm also CC'ing Lincoln who is seeing this issue out in the wild with
> >>> a 5.6.1 client against a 5.5.5 server (see attached). For comparison,
> >>> I've also attached a successful copy for a 5.5.5 client vs 5.5.5
> >>> server for the same file.
> >>>
> >>> Bockjoo: would you be able to try downgrading to a 5.5 client and
> >>> seeing if that resolves your issue?
> >>>
> >>> Thanks,
> >>> Brian
> >>>
> >>>
> >>> On 8/28/23 08:42, Bockjoo Kim wrote:
> >>>> Hi Guilherme,
> >>>>
> >>>> The client machine (xrootd 5.6.1) that I had the issue is a fully
> >>>> credible OSG CE.
> >>>>
> >>>> So, there should be no issue with the CA and X509_CERT_DIR.
> >>>>
> >>>> I have used this python script to reproduce the issue on the client
> >>>> machine:
> >>>>
> >>>> #############################################
> >>>>
> >>>> import os
> >>>> import sys
> >>>> import errno
> >>>> import subprocess
> >>>> import zlib
> >>>> import random
> >>>> from XRootD import client
> >>>> from XRootD.client.flags import OpenFlags
> >>>>
> >>>> ENDPOINT='cmsio2.rc.ufl.edu:1094'
> >>>> SAM_TEST_FILE='/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root'
> >>>>
> >>>>
> >>>> print ("XRootD Client Versin",client.__version__)
> >>>> cmd = [ "xrdfs "+ENDPOINT+" query config version" ]
> >>>> try:
> >>>> result = subprocess.run(cmd, shell=True, capture_output=True,
> >>>> text=True)
> >>>> print("XRootD Server Version", result.stdout)
> >>>> except subprocess.TimeoutExpired:
> >>>> print("connecting to endpoint timed out")
> >>>>
> >>>>
> >>>> os.environ["X509_CERT_DIR"] =
> >>>> "/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates"
> >>>> os.environ["X509_USER_PROXY"] = "/home/bockjoo/.cmsuser.proxy"
> >>>> os.environ["X509_USER_PROXY_NONCMS"] = "/home/bockjoo/.griduser.proxy"
> >>>> os.environ["XRD_NETWORKSTACK"] = "IPv4"
> >>>> with client.File() as f:
> >>>> status, response = f.open("root://" + ENDPOINT + "/" + \
> >>>> SAM_TEST_FILE, flags=OpenFlags.READ, timeout=90)
> >>>> if ( not status.ok ):
> >>>> print (("\nopen(root://%s/%s, flags=OpenFlags.READ,
> >>>> time" + \
> >>>> "out=90)\nXRootDStatus.code=%d \"%s\"\n") % \
> >>>> (ENDPOINT, SAM_TEST_FILE, status.code, \
> >>>> status.message.replace("\n", "")))
> >>>> pass
> >>>> status, data = f.read(offset=0, size=65536, timeout=90)
> >>>> if ( not status.ok ):
> >>>> print(("\n%s\nread(offset=0, size=65536,
> >>>> timeout=90)\n" + \
> >>>> "XRootDStatus.code=%d \"%s\"\n") %
> >>>> (SAM_TEST_FILE, \
> >>>> status.code, status.message.replace("\n",
> >>>> "")))
> >>>> pass
> >>>> print ("Open Status",status.ok)
> >>>>
> >>>> #############################################
> >>>>
> >>>> You can choose the endpoint and the file of your choosing with the
> >>>> 5.5.5 server
> >>>>
> >>>> to test it.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Bockjoo
> >>>>
> >>>> On 8/28/23 09:33, Guilherme Amadio wrote:
> >>>>> Dear Bockjoo,
> >>>>>
> >>>>> On Sat, Aug 26, 2023 at 04:14:28PM -0400, Bockjoo Kim wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am seeing a python XRootD file open issue for the 5.6.1 client
> >>>>>> with a
> >>>>>> 5.5.5 server :
> >>>>>>
> >>>>>> =============================================================
> >>>>>>
> >>>>>> XRootD Client Versin 5.6.1
> >>>>>> XRootD Server Version v5.5.5
> >>>>>>
> >>>>>> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root,
> >>>>>>
> >>>>>> flags=OpenFlags.READ, timeout=90)
> >>>>>> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily
> >>>>>> unavailable: Unable to connect to cmsio2.rc.ufl.edu; error_ssl"
> >>>>>>
> >>>>>> ---------------------------------------------------------------------------
> >>>>>>
> >>>>>> ValueError Traceback (most recent
> >>>>>> call last)
> >>>>>> /tmp/ipykernel_4179061/812350213.py in <module>
> >>>>>> 40 status.message.replace("\n", "")))
> >>>>>> 41 #pass
> >>>>>> ---> 42 status, data = f.read(offset=0, size=65536,
> >>>>>> timeout=90)
> >>>>>> 43 if ( not status.ok ):
> >>>>>> 44 print(("\n%s\nread(offset=0, size=65536,
> >>>>>> timeout=90)\n" + \
> >>>>>>
> >>>>>> /opt/cms/services/anaconda3/lib/python3.9/site-packages/XRootD/client/file.py
> >>>>>>
> >>>>>> in read(self, offset, size, timeout, callback)
> >>>>>> 124 return XRootDStatus(self.__file.read(offset, size,
> >>>>>> timeout, callback))
> >>>>>> 125
> >>>>>> --> 126 status, response = self.__file.read(offset, size, timeout)
> >>>>>> 127 return XRootDStatus(status), response
> >>>>>> 128
> >>>>>>
> >>>>>> ValueError: I/O operation on closed file
> >>>>>>
> >>>>>> ===============================================================
> >>>>>>
> >>>>>> Here, XRootD Server is configured with TLS.
> >>>>>>
> >>>>>> If I remove TLS configuration of the 5.5.5 server, there is no issue.
> >>>>>>
> >>>>>> If I switch to 5.5.5 client, there is no issue.
> >>>>>>
> >>>>>> Is this expected?
> >>>>> It may or may not be. When I wrote the patch, I tested several
> >>>>> scenarios
> >>>>> (see
> >>>>> https://github.com/xrootd/xrootd/pull/2031#issuecomment-1589380486).
> >>>>> The error message that you see is likely caused by a client that cannot
> >>>>> validate the server with TLS (because it does not have the proper CA
> >>>>> certificates installed locally). So I suggest you to try with xrdcp
> >>>>> --notlsok option, or export X509_CERT_DIR=/dev/null to force the client
> >>>>> into not being able to do TLS at all. If the directory
> >>>>> /etc/grid-security
> >>>>> exists on your machine, but the client cannot verify the server, and
> >>>>> TLS
> >>>>> is enforced, then this behavior is expected. Otherwise, please export
> >>>>> XRD_LOGLEVEL=Dump, re-run the command and send us the output so I can
> >>>>> investigate this issue further. You may also want to install the proper
> >>>>> certificates to let the client validate the server to be able to use
> >>>>> TLS.
> >>>>>
> >>>>> Best regards,
> >>>>> -Guilherme
> >>>> ########################################################################
> >>>> Use REPLY-ALL to reply to list
> >>>>
> >>>> To unsubscribe from the XROOTD-L list, click the following link:
> >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
