Using XRootD 4.11.3 and xrdhttpvoms 0.2.5 (not sure if this component is involved!), `gfal-copy` fails to authenticate ~50 % of the time with: ``` gfal-copy error: 13 (Permission denied) - TRANSFER ERROR: Copy failed with mode streamed, with error: Authentication error, reached maximum number of attempts ``` Here's the test command: ``` for A in {1..9}; do gfal-copy file://$(pwd)/1M https://xrootd.physik.uni-bonn.de:1094//cephfs/grid/dteam/of_1M_${A}; done ``` I activated `http.trace all` and extracted the two cases. **Case A (working** ``` 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: received dlen: 16 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: received dump: 22 03 01 02 00 01 00 01 -04 03 03 -19 -99 -18 -54 00 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: This does not look like http at pos 0 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: This may look like https 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Protocol matched. https: 1 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Process. lp:0x7f70200019a8 reqstate: 0 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Setting host: [::ffff:188.184.98.75] 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Entering SSL_accept... 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: SSL_accept returned :1 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: SSL_get_verify_result returned :0 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Extracting auth info. 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: SSL_get_peer_certificate returned :0x7f701c00d5a0 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Subject name is : '/C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth/CN=461443643' 200412 22:56:48 24968 ?:[log in to unmask] sysXrdHttp: Setting link name: 'eyermuth' 200412 22:56:48 24968 eyermuth.0:[log in to unmask] SSL_get_peer_certificate returned :0x7f701c00d5a0 200412 22:56:48 24968 eyermuth.0:[log in to unmask] SSL_get_verify_result returned :0 200412 22:56:48 24968 eyermuth.0:[log in to unmask] SSL_get_peer_cert_chain :0x7f701c000e20 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS data - user: '/C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth' 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS data - vorg: 'dteam' 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS data - fqan[0]:/dteam/Role=NULL/Capability=NULL 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS data - grps: '/dteam /dteam/Role=NULL /dteam/Role=NULL/Capability=NULL' 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS data - role: 'NULL' 200412 22:56:48 24968 eyermuth.0:[log in to unmask] VOMS proxy info - name: '/C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth' VO: dteam grps: '/dteam /dteam/Role=NULL /dteam/Role=NULL/Capability=NULL' 200412 22:56:48 24968 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576 200412 22:56:48 24968 sysXrdHttp: getDataOneShot sslavail: 1048576 200412 22:56:48 24968 sysXrdHttp: read 190 of 1048576 bytes 200412 22:56:48 24968 sysXrdHttp: rc:43 got hdr line: HEAD //cephfs/grid/dteam/of_1M_9 HTTP/1.1 200412 22:56:48 24968 sysXrdHttp: Parsing first line: HEAD //cephfs/grid/dteam/of_1M_9 HTTP/1.1 200412 22:56:48 24968 sysXrdHttp: rc:55 got hdr line: User-Agent: gfal2-util/1.5.3 gfal2/2.17.2 neon/0.0.29 200412 22:56:48 24968 sysXrdHttp: rc:14 got hdr line: Keep-Alive: 200412 22:56:48 24968 sysXrdHttp: rc:24 got hdr line: Connection: Keep-Alive 200412 22:56:48 24968 sysXrdHttp: rc:14 got hdr line: TE: trailers 200412 22:56:48 24968 sysXrdHttp: rc:38 got hdr line: Host: xrootd.physik.uni-bonn.de:1094 200412 22:56:48 24968 sysXrdHttp: rc:2 got hdr line: 200412 22:56:48 24968 sysXrdHttp: rc:2 detected header end. 200412 22:56:48 24968 XrootdBridge: /C=DE/O=.62:[log in to unmask] login as /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth 200412 22:56:48 24968 /C=DE/O=.62:[log in to unmask] sysXrdHttp: Process. lp:0x7f70200019a8 reqstate: 0 200412 22:56:48 24968 /C=DE/O=.62:[log in to unmask] sysXrdHttp: Process is exiting rc:0 200412 22:56:48 24968 acc_Audit: http grant /C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth@[::ffff:188.184.98.75] stat /cephfs/grid/dteam/of_1M_9 ``` **Case B (not working)** ``` 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: received dlen: 16 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: received dump: 22 03 01 14 -98 01 00 14 -102 03 03 -79 08 -12 06 00 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: This does not look like http at pos 0 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: This may look like https 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Protocol matched. https: 1 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Process. lp:0x7f7020000ee8 reqstate: 0 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Setting host: [::ffff:188.184.98.75] 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Entering SSL_accept... 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: SSL_accept returned :1 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: SSL_get_verify_result returned :0 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Extracting auth info. 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: SSL_get_peer_certificate returned :0x7f70100015d0 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Subject name is : '/C=DE/O=GermanGrid/OU=UniBonn/CN=Oliver Freyermuth/CN=461443643' 200412 22:56:48 24985 ?:[log in to unmask] sysXrdHttp: Setting link name: 'eyermuth' 200412 22:56:48 24985 eyermuth.0:[log in to unmask] SSL_get_peer_certificate returned :0x7f70100015d0 200412 22:56:48 24985 eyermuth.0:[log in to unmask] SSL_get_verify_result returned :0 200412 22:56:48 24985 eyermuth.0:[log in to unmask] SSL_get_peer_cert_chain :0 200412 22:56:48 24985 sysXrdHttp: getDataOneShot BuffAvailable: 1048576 maxread: 1048576 200412 22:56:48 24985 sysXrdHttp: getDataOneShot sslavail: 1048576 200412 22:56:48 24985 sysXrdHttp: read 187 of 1048576 bytes 200412 22:56:48 24985 sysXrdHttp: rc:43 got hdr line: HEAD //cephfs/grid/dteam/of_1M_9 HTTP/1.1 200412 22:56:48 24985 sysXrdHttp: Parsing first line: HEAD //cephfs/grid/dteam/of_1M_9 HTTP/1.1 200412 22:56:48 24985 sysXrdHttp: rc:55 got hdr line: User-Agent: gfal2-util/1.5.3 gfal2/2.17.2 neon/0.0.29 200412 22:56:48 24985 sysXrdHttp: rc:14 got hdr line: TE: trailers 200412 22:56:48 24985 sysXrdHttp: rc:38 got hdr line: Host: xrootd.physik.uni-bonn.de:1094 200412 22:56:48 24985 sysXrdHttp: rc:35 got hdr line: Accept: application/metalink4+xml 200412 22:56:48 24985 sysXrdHttp: rc:2 got hdr line: 200412 22:56:48 24985 sysXrdHttp: rc:2 detected header end. 200412 22:56:48 24985 XrootdBridge: Oliver F.63:[log in to unmask] login as Oliver Freyermuth 200412 22:56:48 24985 Oliver F.63:[log in to unmask] sysXrdHttp: Process. lp:0x7f7020000ee8 reqstate: 0 200412 22:56:48 24985 Oliver F.63:[log in to unmask] sysXrdHttp: Process is exiting rc:0 200412 22:56:48 24985 acc_Audit: http deny Oliver Freyermuth@[::ffff:188.184.98.75] stat /cephfs/grid/dteam/of_1M_9 ``` The notable difference is that `SSL_get_peer_cert_chain :0` did not get a chain. Here's our config on the redirector: ``` acc.audit deny grant acc.authdb /etc/xrootd/auth_file-grid acc.authrefresh 60 all.export /cephfs/grid/dteam r/w nostage all.manager xrootd.physik.uni-bonn.de:1213 all.role server all.role manager if xrootd.physik.uni-bonn.de all.sitename UNI-BONN cms.allow host xrootd.physik.uni-bonn.de cms.allow host xrootd*.physik.uni-bonn.de cms.dfs limit 0 lookup distrib mdhold 0 redirect immed retries 2 cms.perf int 60s pgm /usr/share/xrootd/utils/cms_monPerf 30 cms.sched cpu 10 io 60 mem 10 pag 10 runq 10 space 0 fuzz 20 gshr 100 affinity none refreset 3600 cms.trace all -debug http.cadir /etc/grid-security/certificates http.cert /etc/grid-security/hostcert.pem http.exthandler xrdmacaroons libXrdMacaroons.so http.exthandler xrdtpc libXrdHttpTPC.so http.header2cgi Authorization authz http.secretkey REMOVED_FROM_HERE http.key /etc/grid-security/hostkey.pem http.secxtractor /usr/lib64/libXrdHttpVOMS.so if exec xrootd xrd.protocol XrdHttp /usr/lib64/libXrdHttp.so fi if xrootd.physik.uni-bonn.de else http.selfhttps2http no fi if xrootd.physik.uni-bonn.de http.desthttps yes fi macaroons.secretkey /etc/xrootd/macaroon-secret ofs.authlib libXrdMacaroons.so ofs.authorize 1 ofs.notify mkdir create |/usr/local/bin/xrootd-eventstream-permission-fix ofs.tpc fcreds ?gsi =X509_USER_PROXY autorm ttl 180 1800 pgm /usr/local/bin/xrdcp-voms --server sec.protocol /usr/lib64 gsi -dlgpxy:1 -exppxy:=creds -ca:1 -crl:3 -gridmap:/dev/null -cert:/etc/grid-security/hostcert.pem -key:/etc/grid-security/hostkey.pem -certdir:/etc/grid-security/certificates -vomsfun:/usr/lib64/libXrdSecgsiVOMS.so -vomsfunparms:certfmt=raw|vos=atlas,ops,dteam,wlcg|grps=/atlas,/ops,/dteam,/wlcg xrd.port 1094 xrd.timeout hail 30 idle 0 kill 10 read 30 xrd.trace conn xrootd.chksum adler32 crc32 md5 xrootd.seclib /usr/lib64/libXrdSec.so ``` (only some exports and the secretkey were removed). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/1177 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1