Hi Guilherme,
I am not sure how to pass XrdSecDEBUG=1 and XrdSecPROTOCOL=gsi to the
python script.
I tried to set them through os.environ, but the output is not verbose.
Below are the xrdfs command line outputs, though. Both works.
Thanks
Bockjoo
[1] at CERN
-bash-4.2$ rpm -qf $(which xrdfs)
xrootd-client-5.6.1-1.el7.x86_64
-bash-4.2$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
xroots://cmsio2.rc.ufl.edu query config version
sec_Client: protocol request for host cmsio2.rc.ufl.edu
token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:Ga�'
sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
Secgsi -------------------------------------------------------------------
Secgsi Mode: client
Secgsi Debug: 1
Secgsi CA dir: /etc/grid-security/certificates/
Secgsi CA verification level: verifyss
Secgsi CRL dir: /etc/grid-security/certificates/
Secgsi CRL extension: .r0
Secgsi CRL check level: try
Secgsi CRL refresh time: 86400
Secgsi Certificate: /afs/cern.ch/user/b/bockjoo/.globus/usercert.pem
Secgsi Key: /afs/cern.ch/user/b/bockjoo/.globus/userkey.pem
Secgsi Proxy file: /afs/cern.ch/user/b/bockjoo/.proxy
Secgsi Proxy validity: 12:00
Secgsi Proxy dep length: 0
Secgsi Proxy bits: 512
Secgsi Proxy sign option: 1
Secgsi Proxy delegation option: 0
Secgsi Pure Cert/Key authentication allowed
Secgsi Allowed server names: [*/]<target host name>[/*]
Secgsi Crypto modules: ssl
Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
Secgsi MDigests: sha256
Secgsi Trusting DNS for hostname checking
Secgsi -------------------------------------------------------------------
sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
230904 18:04:27 4963 secgsi_VerifyCA: Warning: CA certificate not
self-signed and integrity not checked: assuming OK (ba240aa8.0)
230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 8 extensions
230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 5 extensions
230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 12 extensions
230904 18:04:27 4963 cryptossl_X509::CertType: certificate has 10 extensions
v5.5.5
[2] Florida
[bockjoo@cms site-packages]$ rpm -qf $(which xrdfs)
xrootd-client-5.5.5-1.2.osg36.el8.x86_64
[bockjoo@cms site-packages]$ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs
xroots://cmsio2.rc.ufl.edu query config version
sec_Client: protocol request for host cmsio2.rc.ufl.edu
token='&P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:'
sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
Secgsi -------------------------------------------------------------------
Secgsi Mode: client
Secgsi Debug: 1
Secgsi CA dir: /etc/grid-security/certificates/
Secgsi CA verification level: verifyss
Secgsi CRL dir: /etc/grid-security/certificates/
Secgsi CRL extension: .r0
Secgsi CRL check level: try
Secgsi CRL refresh time: 86400
Secgsi Certificate: /home/bockjoo/.globus/usercert.pem
Secgsi Key: /home/bockjoo/.globus/userkey.pem
Secgsi Proxy file: /home/bockjoo/.cmsuser.proxy
Secgsi Proxy validity: 12:00
Secgsi Proxy dep length: 0
Secgsi Proxy bits: 512
Secgsi Proxy sign option: 1
Secgsi Proxy delegation option: 0
Secgsi Pure Cert/Key authentication allowed
Secgsi Allowed server names: [*/]<target host name>[/*]
Secgsi Crypto modules: ssl
Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
Secgsi MDigests: sha1:md5
Secgsi Trusting DNS for hostname checking
Secgsi -------------------------------------------------------------------
sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0'
230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
extensions
230904 12:10:43 1107303 secgsi_VerifyCA: Warning: CA certificate not
self-signed and integrity not checked: assuming OK (ba240aa8.0)
230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 8
extensions
230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 5
extensions
230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 12
extensions
230904 12:10:43 1107303 cryptossl_X509::CertType: certificate has 10
extensions
v5.5.5
On 9/4/23 09:53, Guilherme Amadio wrote:
> Dear Bokjoo,
>
> I tried your script, and I can connect without problems to 5.5 servers
> here at CERN where I have access via gsi protocol. I get permission denied
> on your server, but no TLS error. I tried both from CentOS 7 (OpenSSL
> 1.0.x) and from my own machine (where I use OpenSSL 1.1.x) with client v5.6.1.
>
> $ XrdSecDEBUG=1 XrdSecPROTOCOL=gsi xrdfs xroots://eoscms.cern.ch query config version
> sec_Client: protocol request for host eoscms.cern.ch token='&P=krb5,[log in to unmask]&P=gsi,v:10600,c:ssl,ca:5168735f.0|4339b4bc.0&P=sss,0.+13:/etc/eos.keytab&P=unix'
> sec_PM: Skipping krb5 only want gsi
> sec_PM: Loaded gsi protocol object from libXrdSecgsi.so
> Secgsi -------------------------------------------------------------------
> Secgsi Mode: client
> Secgsi Debug: 1
> Secgsi CA dir: /etc/grid-security/certificates/
> Secgsi CA verification level: verifyss
> Secgsi CRL dir: /etc/grid-security/certificates/
> Secgsi CRL extension: .r0
> Secgsi CRL check level: try
> Secgsi CRL refresh time: 86400
> Secgsi Certificate: /home/amadio/.globus/usercert.pem
> Secgsi Key: /home/amadio/.globus/userkey.pem
> Secgsi Proxy file: /tmp/x509up_u75748
> Secgsi Proxy validity: 12:00
> Secgsi Proxy dep length: 0
> Secgsi Proxy bits: 512
> Secgsi Proxy sign option: 1
> Secgsi Proxy delegation option: 0
> Secgsi Pure Cert/Key authentication allowed
> Secgsi Allowed server names: [*/]<target host name>[/*]
> Secgsi Crypto modules: ssl
> Secgsi Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc
> Secgsi MDigests: sha256
> Secgsi Trusting DNS for hostname checking
> Secgsi -------------------------------------------------------------------
> sec_PM: Using gsi protocol, args='v:10600,c:ssl,ca:5168735f.0|4339b4bc.0'
> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> 230904 10:30:53 125169 secgsi_VerifyCA: Warning: CA certificate not self-signed and integrity not checked: assuming OK (5168735f.0)
> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 4 extensions
> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 12 extensions
> 230904 10:30:53 125169 cryptossl_X509::CertType: certificate has 10 extensions
> 5.5.10
>
> Could you please post a verbose output of the error you see, like I did
> above for the working case? I think that the client might be missing
> some configuration or some certificates which are required to validate
> the server and that's why you see errors. With XrdSecDEBUG=1 XrdSecPROTOCOL=gsi
> we should be able to get enough information to debug this.
>
> Best regards,
> -Guilherme
>
> On Tue, Aug 29, 2023 at 12:39:40PM -0400, Bockjoo Kim wrote:
>> Yes, as I posted in the original email: "If I switch to 5.5.5 client, there is no issue"
>> Thanks,
>> Bockjoo
>>
>> On 8/29/23 12:11, Brian Lin wrote:
>>> Hi all,
>>>
>>> We're currently seeing similar TLS issues in osg-test with xrdcp. It's
>>> why we haven't released 5.6.x in the OSG repos yet:
>>> https://opensciencegrid.atlassian.net/browse/SOFTWARE-5623?focusedCommentId=380033.
>>>
>>> I'm also CC'ing Lincoln who is seeing this issue out in the wild with
>>> a 5.6.1 client against a 5.5.5 server (see attached). For comparison,
>>> I've also attached a successful copy for a 5.5.5 client vs 5.5.5
>>> server for the same file.
>>>
>>> Bockjoo: would you be able to try downgrading to a 5.5 client and
>>> seeing if that resolves your issue?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>> On 8/28/23 08:42, Bockjoo Kim wrote:
>>>> Hi Guilherme,
>>>>
>>>> The client machine (xrootd 5.6.1) that I had the issue is a fully
>>>> credible OSG CE.
>>>>
>>>> So, there should be no issue with the CA and X509_CERT_DIR.
>>>>
>>>> I have used this python script to reproduce the issue on the client
>>>> machine:
>>>>
>>>> #############################################
>>>>
>>>> import os
>>>> import sys
>>>> import errno
>>>> import subprocess
>>>> import zlib
>>>> import random
>>>> from XRootD import client
>>>> from XRootD.client.flags import OpenFlags
>>>>
>>>> ENDPOINT='cmsio2.rc.ufl.edu:1094'
>>>> SAM_TEST_FILE='/store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root'
>>>>
>>>>
>>>> print ("XRootD Client Versin",client.__version__)
>>>> cmd = [ "xrdfs "+ENDPOINT+" query config version" ]
>>>> try:
>>>> result = subprocess.run(cmd, shell=True, capture_output=True,
>>>> text=True)
>>>> print("XRootD Server Version", result.stdout)
>>>> except subprocess.TimeoutExpired:
>>>> print("connecting to endpoint timed out")
>>>>
>>>>
>>>> os.environ["X509_CERT_DIR"] =
>>>> "/cvmfs/cms.cern.ch/grid/etc/grid-security/certificates"
>>>> os.environ["X509_USER_PROXY"] = "/home/bockjoo/.cmsuser.proxy"
>>>> os.environ["X509_USER_PROXY_NONCMS"] = "/home/bockjoo/.griduser.proxy"
>>>> os.environ["XRD_NETWORKSTACK"] = "IPv4"
>>>> with client.File() as f:
>>>> status, response = f.open("root://" + ENDPOINT + "/" + \
>>>> SAM_TEST_FILE, flags=OpenFlags.READ, timeout=90)
>>>> if ( not status.ok ):
>>>> print (("\nopen(root://%s/%s, flags=OpenFlags.READ,
>>>> time" + \
>>>> "out=90)\nXRootDStatus.code=%d \"%s\"\n") % \
>>>> (ENDPOINT, SAM_TEST_FILE, status.code, \
>>>> status.message.replace("\n", "")))
>>>> pass
>>>> status, data = f.read(offset=0, size=65536, timeout=90)
>>>> if ( not status.ok ):
>>>> print(("\n%s\nread(offset=0, size=65536,
>>>> timeout=90)\n" + \
>>>> "XRootDStatus.code=%d \"%s\"\n") %
>>>> (SAM_TEST_FILE, \
>>>> status.code, status.message.replace("\n",
>>>> "")))
>>>> pass
>>>> print ("Open Status",status.ok)
>>>>
>>>> #############################################
>>>>
>>>> You can choose the endpoint and the file of your choosing with the
>>>> 5.5.5 server
>>>>
>>>> to test it.
>>>>
>>>> Thanks,
>>>>
>>>> Bockjoo
>>>>
>>>> On 8/28/23 09:33, Guilherme Amadio wrote:
>>>>> Dear Bockjoo,
>>>>>
>>>>> On Sat, Aug 26, 2023 at 04:14:28PM -0400, Bockjoo Kim wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am seeing a python XRootD file open issue for the 5.6.1 client
>>>>>> with a
>>>>>> 5.5.5 server :
>>>>>>
>>>>>> =============================================================
>>>>>>
>>>>>> XRootD Client Versin 5.6.1
>>>>>> XRootD Server Version v5.5.5
>>>>>>
>>>>>> open(root://cmsio2.rc.ufl.edu:1094//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/A64CCCF2-5C76-E711-B359-0CC47A78A3F8.root,
>>>>>>
>>>>>> flags=OpenFlags.READ, timeout=90)
>>>>>> XRootDStatus.code=110 "[FATAL] TLS error: resource temporarily
>>>>>> unavailable: Unable to connect to cmsio2.rc.ufl.edu; error_ssl"
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>> ValueError Traceback (most recent
>>>>>> call last)
>>>>>> /tmp/ipykernel_4179061/812350213.py in <module>
>>>>>> 40 status.message.replace("\n", "")))
>>>>>> 41 #pass
>>>>>> ---> 42 status, data = f.read(offset=0, size=65536,
>>>>>> timeout=90)
>>>>>> 43 if ( not status.ok ):
>>>>>> 44 print(("\n%s\nread(offset=0, size=65536,
>>>>>> timeout=90)\n" + \
>>>>>>
>>>>>> /opt/cms/services/anaconda3/lib/python3.9/site-packages/XRootD/client/file.py
>>>>>>
>>>>>> in read(self, offset, size, timeout, callback)
>>>>>> 124 return XRootDStatus(self.__file.read(offset, size,
>>>>>> timeout, callback))
>>>>>> 125
>>>>>> --> 126 status, response = self.__file.read(offset, size, timeout)
>>>>>> 127 return XRootDStatus(status), response
>>>>>> 128
>>>>>>
>>>>>> ValueError: I/O operation on closed file
>>>>>>
>>>>>> ===============================================================
>>>>>>
>>>>>> Here, XRootD Server is configured with TLS.
>>>>>>
>>>>>> If I remove TLS configuration of the 5.5.5 server, there is no issue.
>>>>>>
>>>>>> If I switch to 5.5.5 client, there is no issue.
>>>>>>
>>>>>> Is this expected?
>>>>> It may or may not be. When I wrote the patch, I tested several
>>>>> scenarios
>>>>> (see
>>>>> https://github.com/xrootd/xrootd/pull/2031#issuecomment-1589380486).
>>>>> The error message that you see is likely caused by a client that cannot
>>>>> validate the server with TLS (because it does not have the proper CA
>>>>> certificates installed locally). So I suggest you to try with xrdcp
>>>>> --notlsok option, or export X509_CERT_DIR=/dev/null to force the client
>>>>> into not being able to do TLS at all. If the directory
>>>>> /etc/grid-security
>>>>> exists on your machine, but the client cannot verify the server, and
>>>>> TLS
>>>>> is enforced, then this behavior is expected. Otherwise, please export
>>>>> XRD_LOGLEVEL=Dump, re-run the command and send us the output so I can
>>>>> investigate this issue further. You may also want to install the proper
>>>>> certificates to let the client validate the server to be able to use
>>>>> TLS.
>>>>>
>>>>> Best regards,
>>>>> -Guilherme
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
