 |
 |
We have a EOS storage system for 3 different experiments (CMS, Belle and ALICE).
Recently we got notified that some CMS test jobs (HammerCloud) are failing to access datasets on our storage system and fail back to accessing the data from a remote storage system.
After looking into this we realized that the user under which the test job is running is wrongly mapped to nobody which has no access on the datasets.
We further looked in to the EOS logs and it looks like that the VOMS parsing/extraction fails for those requests:
201214 03:27:33 60328 secgsi_ServerDoCert: no signed DH parameters from client:grid.cms.307:555@[::ffff:172.24.77.37] : will not delegate x509 proxy to it
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 2 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 2 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 5 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:27:33 60328 cryptossl_X509::CertType: certificate has 11 extensions
201214 03:27:33 60328 secgsi_XrdOucGMap::dn2user: no valid match found for DN '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba'
201214 03:27:33 60328 secgsi_Authenticate: WARNING: user mapping lookup failed - use DN or DN-hash as name
201214 03:27:33 60328 secgsi_ExtractVOMS: No VOMS attributes in proxy chain
201214 03:27:33 60328 secgsi_Authenticate: VOMS: Entity.vorg: <none>
201214 03:27:33 60328 secgsi_Authenticate: VOMS: Entity.grps: <none>
201214 03:27:33 60328 secgsi_Authenticate: VOMS: Entity.role: <none>
201214 03:27:33 60328 secgsi_Authenticate: VOMS: Entity.endorsements: <none>
201214 03:27:33 60328 XrootdXeq: grid.cms.307:555@[::ffff:172.24.77.37] pvt IPv4 login as /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba
Interestingly for other request by the same user the VOMS extension can be parsed/extracted and the user is properly mapped
201214 03:29:45 60177 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:29:45 60177 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:29:45 60177 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:29:45 60177 cryptossl_X509::CertType: certificate has 3 extensions
201214 03:29:45 60177 cryptossl_X509::CertType: certificate has 11 extensions
201214 03:29:45 60177 secgsi_XrdOucGMap::dn2user: no valid match found for DN '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba'
201214 03:29:45 60177 secgsi_Authenticate: WARNING: user mapping lookup failed - use DN or DN-hash as name
201214 03:29:45 60177 secgsi_Authenticate: VOMS: Entity.vorg: cms
201214 03:29:45 60177 secgsi_Authenticate: VOMS: Entity.grps: /cms/GGUSExpert
201214 03:29:45 60177 secgsi_Authenticate: VOMS: Entity.role: production
201214 03:29:45 60177 secgsi_Authenticate: VOMS: Entity.endorsements: /cms/Role=production/Capability=NULL,/cms/ALARM/Role=NULL/Capability=NULL,/cms/GGUSExpert/Role=NULL/Capability=NULL,/cms/Role=NULL/Capability=NULL,/cms/TEAM/Role=NULL/Capability=NULL
201214 03:29:45 60177 XrootdXeq: etf.1322408:[log in to unmask] pub IPv4 login as /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba
201214 03:29:45 time=1607912985.146474 func=IdMap level=INFO logid=static.............................. [log in to unmask]:1094 tid=00007ff81f9fc700 source=Mapping:993 tident= sec=(null) uid=99 gid=99 name=- geo="" sec.prot=gsi sec.name="/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba" sec.host="etf-01.cern.ch" sec.vorg="cms" sec.grps="/cms/GGUSExpert" sec.role="production" sec.info="/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba" sec.app="" sec.tident="etf.1322408:[log in to unmask]" vid.uid=43349 vid.gid=43350
The log for the entire test job can be found here
Below is the proxy cert information from the log above:
======== PROXY INFORMATION START at Fri Dec 11 12:11:23 GMT 2020 ========
subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba/CN=269832904/CN=249568208/CN=788980879/CN=3177786675/CN=720700430/CN=2034580094
issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba/CN=269832904/CN=249568208/CN=788980879/CN=3177786675/CN=720700430
identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba/CN=269832904/CN=249568208/CN=788980879/CN=3177786675/CN=720700430
type : RFC compliant proxy
strength : 1024 bits
path : /srv/db85de4154ab6954b8b313bee9b26408357e57f4
timeleft : 23:59:58
key usage : Digital Signature, Key Encipherment
=== VO cms extension information ===
VO : cms
subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba
issuer : /DC=ch/DC=cern/OU=computers/CN=lcg-voms2.cern.ch
attribute : /cms/Role=production/Capability=NULL
attribute : /cms/ALARM/Role=NULL/Capability=NULL
attribute : /cms/GGUSExpert/Role=NULL/Capability=NULL
attribute : /cms/Role=NULL/Capability=NULL
attribute : /cms/TEAM/Role=NULL/Capability=NULL
attribute : Monuserid = iL5reB0c9R (cms)
timeleft : 141:06:14
uri : lcg-voms2.cern.ch:15002
======== PROXY INFORMATION FINISH at Fri Dec 11 12:11:23 GMT 2020 ========
The EOS mgm nodes have following security config:
sec.protocol gsi -cert:/etc/grid-security/daemon/mgm-1.eos.grid.vbc.ac.at.crt -key:/etc/grid-security/daemon/mgm-1.eos.grid.vbc.ac.at.key -gridmap:/etc/grid-security/grid-mapfile -crl:1 -d:1 -gmapopt:11 -gmapto:60 -vomsat:1 -moninfo:1 -exppxy:/var/eos/auth/gsi#<uid>
We are using following xrootd version:
[root@mgm-1 ~]# rpm -qa | grep xrootd
eos-xrootd-4.12.5-1.el7.cern.x86_64
xrootd-alicetokenacc-1.3.1-1.x86_64
xrootd-4.12.5-1.el7.x86_64
xrootd-client-libs-4.12.5-1.el7.x86_64
xrootd-server-libs-4.12.5-1.el7.x86_64
xrootd-libs-4.12.5-1.el7.x86_64
xrootd-server-4.12.5-1.el7.x86_64
xrootd-voms-4.12.5-1.el7.x86_64
xrootd-selinux-4.12.5-1.el7.noarch
We already posted this in the EOS forum but to us it looks like EOS is doing the right thing, however something goes wrong when parsing the VOMS extension of the proxy cert which might be an xrootd issue.
We would appreciate any pointers on how to solve the issue or further debug it
thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1