Print

Print
Thanks Fabio,

Which version of XRootD are you running?
I’ll look deeper into your configuration and try to figure out if something is missing on my side. 

Cheers,
Dejan


On 8 Feb 2023, at 16:33, Fabio Andrijauskas <[log in to unmask]> wrote:

Hi,

  I can see any obvious issue, this config is working for me:


all.adminpath /tmp/xrootd/var/spool
all.pidpath   /tmp/xrootd/var/run

 

oss.localroot /origin
all.export /

 

pfc.trace all
ofs.trace all
xrd.trace all -sched
pss.setopt DebugLevel 5
scitokens.trace all

 

 

# Enable checksum
xrootd.chksum adler32

 

# Config TLS
xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
xrootd.tls capable all -data

 

sec.level all compatible
all.sitename ucsd
# Enable Security
xrootd.seclib libXrdSec.so

 

# Enable "gsi" security
#sec.protparm gsi -vomsfun:libXrdVoms.so -vomsfunparms:dbg
#sec.protocol gsi -ca:1 -crl:3 -gridmap:/dev/null

 

macaroons.secretkey /etc/xrootd/macaroon-secret

 

# Authorizaton
acc.audit deny
acc.authdb /etc/xrootd/auth_file
acc.authrefresh 60
ofs.authorize 1

 

# Xrootd TPC using rendezvous key
ofs.tpc logok autorm pgm /etc/xrootd/xrdcp-tpc.sh

 

# Env var needed by the above TPC script.
setenv X509_USER_CERT = /etc/grid-security/xrd/xrdcert.pem 
setenv X509_USER_KEY = /etc/grid-security/xrd/xrdkey.pem 

 

 

 

 

ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
ofs.authlib ++ libXrdMacaroons.so

 

xrd.protocol http libXrdHttp.so
http.exthandler xrdtpc libXrdHttpTPC.so

 

 

 

cat /etc/xrootd/macaroon-secret
SPKWus+p1S/dSpk15W9Cu/hCWeM0LnPuiNItzyAhkgUlUkAvzRYSOloI2HCSLKvk
HzWOu3pTjlx1SsG2nyEyCw==

 

 cat /etc/xrootd/auth_file
g /xrootd /data rl

 

 

 

cat /etc/xrootd/scitokens.cfg
[Global]
onmissing = passthrough

 

[Issuer OSG Monitoring]
base_path = /
map_subject = false
default_user = xrootd


cat  /etc/xrootd/xrdcp-tpc.sh
#!/bin/sh
set -- `getopt S: -S 1 $*`
while [ $# -gt 0 ]
do
  case $1 in
  -S)
      ((nstreams=$2-1))
      [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams"
      shift 2
      ;;
  --)
      shift
      break
      ;;
  esac
done

 

src=$1
dst=$2
xrdcp --server $TCPstreamOpts -f $src root://$XRDXROOTD_PROXY/${dst}

On Wed, Feb 8, 2023 at 5:34 AM Dejan Vitlacil <[log in to unmask]> wrote:
Hi Fabio,

Thanks for reaching out!

This is command I’m using:
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT"   --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo

If I comment out “ofs.authorize” - there are no problems in uploading a file:
#############################################################################
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_NOauthz.repo
* We are completely uploaded and fine
* Closing connection 0
:-)
[centos@xrootd ~ ls -lh /data
total 16K
drwxr-xr-x. 2 xrootd xrootd   6 Feb  6 14:13 escape
-rw-r--r--. 1 xrootd xrootd  77 Jan 25 13:21 four.txt
-rw-r--r--. 1 xrootd xrootd 168 Feb  6 15:22 testfile-token.repo
-rw-r--r--. 1 xrootd xrootd 168 Feb  8 13:17 testfile-token_NOauthz.repo
-rw-r--r--. 1 xrootd xrootd 168 Feb  2 14:18 testfile.repo
[centos@xrootd ~]$ 
##############################################################################

I also added more tracing as you suggested.

Cheers,
Dejan

###############################################################################
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT"   --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo
* About to connect() to xrootd.e-commons.chalmers.se port 80 (#0)
*   Trying ::1...
* Connected to xrootd.e-commons.chalmers.se (::1) port 80 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: fullchain.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* start date: Jan 30 14:29:24 2023 GMT
* expire date: Apr 30 14:29:23 2023 GMT
* issuer: CN=R3,O=Let's Encrypt,C=US
> PUT /data/testfile-token_new.repo HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Authorization: Bearer eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjJkZWJhOWQxLTk1NTgtNDk2My05NWJjLTc1ZTk5M2UzYzgyZSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2FueSIsIm5iZiI6MTY3NTg1OTE2Mywic2NvcGUiOiJhZGRyZXNzIG9wZW5pZCBwcm9maWxlIHN0b3JhZ2Uuc3RhZ2U6XC8gZWR1cGVyc29uX2VudGl0bGVtZW50IHBob25lIG9mZmxpbmVfYWNjZXNzIGVkdXBlcnNvbl9zY29wZWRfYWZmaWxpYXRpb24gZWR1cGVyc29uX2Fzc3VyYW5jZSBlbWFpbCB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC9pYW0tZXNjYXBlLmNsb3VkLmNuYWYuaW5mbi5pdFwvIiwiZXhwIjoxNjc1ODYyNzYzLCJpYXQiOjE2NzU4NTkxNjMsImp0aSI6ImIyZTZjOGU0LTZlM2ItNDkzZS04ODcwLWNjNDY0NWYzMTk0OSIsImNsaWVudF9pZCI6ImMyNjU0ZWZiLWY4MTUtNGE2ZS04NDQ4LWIyZWU4MGVjYjE2YiIsIndsY2cuZ3JvdXBzIjpbIlwvZXNjYXBlIiwiXC9lc2NhcGVcL3NrYSJdfQ.L2DKM95W1ovF3QOIPrYR5ifGkyXlDgW2FwiKoSFm2XXAXVdzqrK36gQBCTu2hqoXaP9-6eU_a6Un0jXaY4Gi457HPUk4mDy8Mm0ZctaWAzOZnMyIIbvv0VKmEfFUDq_gBLMr1Lq2PbIuvHbhGhi58dyNlj4pdI8Ped19Q4fNXzg
> Content-Length: 168
> Expect: 100-continue
> 
< HTTP/1.1 403 Forbidden
< Connection: Keep-Alive
< Server: XrootD/v5.5.1
< Content-Length: 66
* HTTP error before end of send, stop sending
< 
Unable to create /data/testfile-token_new.repo; permission denied
* Closing connection 0
[centos@xrootd ~]$ 
#######################################################################

[centos@xrootd ~]$ sudo cat /etc/xrootd/scitokens.cfg 
[Global]
onmissing = passthrough
# don't use https://wlcg.cern.ch/jwt/v1/any on production instances

[Issuer ESCAPE IAM]
base_path = /data
map_subject = false
default_user = xrootd
[centos@xrootd ~]$ 

########################################################################

230208 13:32:12 22541 Starting on Linux 3.10.0-1160.81.1.el7.x86_64
230208 13:32:12 22541 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-http.cfg -k fifo -s /run/xrootd/xrootd-http.pid -n http
Copr.  2004-2012 Stanford University, xrd version v5.5.1
Config warning: this hostname, localhost, is registered without a domain qualification.
++++++ xrootd http@localhost initialization started.
Config using configuration file /etc/xrootd/xrootd-http.cfg
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.protocol XrdHttp:80 libXrdHttp.so 
=====> xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
=====> xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
=====> xrd.trace all -sched
=====> continue /etc/xrootd/config.d/
++++++ xrootd http@localhost TLS initialization started.
------ xrootd http@localhost TLS initialization ended.
Config maximum number of connections restricted to 65536
Config maximum number of threads restricted to 7149
230208 13:32:12 22541 Xrd_Config: sendfile enabled.
230208 13:32:12 22541 Xrd_LinkCtl: Allocating 64 link objects at a time
230208 13:32:12 22541 Xrd_Poll: Starting poller 0
230208 13:32:12 22541 Xrd_Poll: Starting poller 1
230208 13:32:12 22541 Xrd_Poll: Starting poller 2
230208 13:32:12 22541 Xrd_ProtLoad: protocol xroot wants to use port 1094
Plugin loaded xrdhttp v5.5.1 from protocol libXrdHttp-5.so
230208 13:32:12 22541 Xrd_ProtLoad: protocol XrdHttp wants to use port 80
230208 13:32:12 22541 Xrd_Config: xroot:1094 wsz=87380
230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object xroot
Copr.  2012 Stanford University, xroot protocol 5.1.0 version v5.5.1
++++++ xroot protocol initialization started.
=====> all.export /data
=====> xrootd.tls capable all -data
=====> xrootd.seclib libXrdSec.so
=====> continue /etc/xrootd/config.d/
Config exporting /data
Plugin loaded secprot v5.5.1 from seclib libXrdSec-5.so
++++++ Authentication system initialization started.
Plugin loaded secztn v5.5.1 from sec.protocol libXrdSecztn-5.so
=====> sec.protocol ztn
=====> continue /etc/xrootd/config.d/
Config 1 authentication directives processed in /etc/xrootd/xrootd-http.cfg
------ Authentication system initialization completed.
++++++ Protection system initialization started.
Config warning: Security level is set to none; request protection disabled!
Config Local  protection level: none
Config Remote protection level: none
------ Protection system initialization completed.
Config Routing for [::1]: local pub4 prv4 pub6 prv6
Config Route all4: 127.0.0.1 Dest=[::127.0.0.1]:1094
Config Route all6: [::1] Dest=[::1]:1094
++++++ File system initialization started.
=====> ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg 
=====> ofs.trace all
=====> continue /etc/xrootd/config.d/
++++++ Storage system initialization started.
=====> all.export /data
=====> continue /etc/xrootd/config.d/
Config effective /etc/xrootd/xrootd-http.cfg oss configuration:
       oss.alloc        0 0 0
       oss.spacescan    600
       oss.fdlimit      32768 65536
       oss.maxsize      0
       oss.trace        0
       oss.xfr          1 deny 10800 keep 1200
       oss.memfile off  max 963475456
       oss.defaults  r/w nocheck nodread nomig nopurge norcreate nostage
       oss.path /data r/w nocheck nodread nomig nopurge norcreate nostage
------ Storage system initialization completed.
++++++ Authorization system initialization started.
230208 13:32:12 22541 acc_Config: Authorization system using configuration in /etc/xrootd/xrootd-http.cfg
=====> acc.authdb /etc/xrootd/Authfile
=====> continue /etc/xrootd/config.d/
Config 1 authorization directives processed in /etc/xrootd/xrootd-http.cfg
Config 1 auth entries processed in /etc/xrootd/Authfile
------ Authorization system initialization completed.
Plugin loaded XrdAccSciTokens v5.5.1 from authlib libXrdAccSciTokens-5.so
++++++ XrdAccSciTokens: Initialized SciTokens-based authorization.
=====> scitokens.trace all 
=====> continue /etc/xrootd/config.d/
230208 13:32:12 22541 scitokens_Config: Logging levels enabled - all
230208 13:32:12 22541 scitokens_Reconfig: Parsing configuration file: /etc/xrootd/scitokens.cfg
230208 13:32:12 22541 scitokens_Reconfig: Configuring issuer https://iam-escape.cloud.cnaf.infn.it/
++++++ Checkpoint initialization started.
++++++ Checkpoint initialization completed.
Config effective /etc/xrootd/xrootd-http.cfg ofs configuration:
       all.role server
       ofs.authorize
       ofs.maxdelay   60
       ofs.persist    manual hold 600 logdir /var/spool/xrootd/http/.ofs/posc.log
       ofs.trace      ffff
       ofs.authlib default 
       ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
------ File system server initialization completed.
Config asynchronous I/O has been disabled!
230208 13:32:12 22541  ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
230208 13:32:12 22541 Xrd_ProtLoad: enabling port 1094 for protocol xroot
230208 13:32:12 22541 Xrd_Config: XrdHttp:80 wsz=87380
230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object XrdHttp
Copr. 2012 CERN IT, an HTTP implementation for the XRootD framework.
++++++ HTTP protocol initialization started.
=====> http.header2cgi Authorization authz
=====> continue /etc/xrootd/config.d/
Config Using xrd.tls to supply 'cert' and 'key'.
Config Using xrd.tlsca to supply 'cadir'.
++++++ HTTPS initialization started.
------ HTTPS initialization completed.
230208 13:32:12 22541 sysConfig: XRDROLE:  server
230208 13:32:12 22541 sysConfig: Configured as HTTP(s) data server.
------ HTTP protocol initialization completed.
230208 13:32:12 22541 Xrd_ProtLoad: enabling port 80 for protocol XrdHttp
------ xrootd http@localhost:80 initialization completed.
230208 13:32:12 22558 TLS_Refresh: CRL refresh started.
230208 13:32:12 22558 TLS_Refresh: CRL refresh will happen in 28800 seconds.
230208 13:32:12 22550 TLS_Refresh: CRL refresh started.
230208 13:32:12 22550 TLS_Refresh: CRL refresh will happen in 28800 seconds.
230208 13:32:32 22546 Xrd_Inet: Accepted connection on port 80 from 25@localhost
230208 13:32:32 22546 Xrd_ProtLoad: matched port 80 protocol XrdHttp
230208 13:32:32 22546 anon.0:25@localhost Xrd_Poll: FD 25 attached to poller 0; num=1
230208 13:32:32 22546 XrootdBridge: unknown.1:25@localhost login as nobody
230208 13:32:32 22546 unknown.1:25@localhost ofs_open: 200-40664 fn=/data/testfile-token_WITHauthz.repo
230208 13:32:32 22546 scitokens_Access: Trying token-based access control
230208 13:32:32 22546 scitokens_Access: Token not found in recent cache; parsing.
230208 13:32:32 22546 scitokens_Access: New valid token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
230208 13:32:32 22546 scitokens_Access: Trying token-based access control
230208 13:32:32 22546 scitokens_Access: Cached token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
230208 13:32:32 22546 ofs_open: unknown.1:25@localhost Unable to create /data/testfile-token_WITHauthz.repo; permission denied
230208 13:32:32 22546 unknown.1:25@localhost ofs_close: use=0 fn=dummy
230208 13:32:32 22546 XrootdXeq: unknown.1:25@localhost disc 0:00:00 (send failure)
230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: Poller 0 removing FD 25
230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: FD 25 detached from poller 0; num=0
[centos@xrootd ~]$ 


On 8 Feb 2023, at 12:42, Fabio Andrijauskas <[log in to unmask]> wrote:

Hi,

 Can you send how are you requesting/creating the files?

   Try to add this to get all the possible logs:

pfc.trace all
ofs.trace all
xrd.trace all -sched
pss.setopt DebugLevel 5
scitokens.trace all

  Did you try to create the the path on the export path?

   Can you send your sci token config file?

   


On Wed, Feb 8, 2023 at 2:25 AM Dejan Vitlacil <[log in to unmask]> wrote:
Hi Matt,

I run out of ideas, so all suggestions are appreciated.
I think unix bits are in place.

Cheers,
Dejan

……
[centos@xrootd ~]$ ls -lh / |grep data
drwxr-xr-x.   3 xrootd xrootd   84 Feb  6 15:22 data
[centos@xrootd ~]$ ls -lh /data
total 12K
drwxr-xr-x. 2 xrootd xrootd   6 Feb  6 14:13 escape
-rw-r--r--. 1 xrootd xrootd  77 Jan 25 13:21 four.txt
-rw-r--r--. 1 xrootd xrootd 168 Feb  6 15:22 testfile-token.repo
-rw-r--r--. 1 xrootd xrootd 168 Feb  2 14:18 testfile.repo
[centos@xrootd ~]$


> On 8 Feb 2023, at 11:05, Doidge, Matt <[log in to unmask]> wrote:
>
> Hello,
> This is a bit of a low-level suggestion, but can the xrootd unix user write to  /data? I had similar looking issues with a test xrootd server and a known working config where I had forgotten to chown the exported path.
>
> My apologies for the noise if you've already checked this.
>
> Cheers,
> Matt
>
> ________________________________________
> From: [log in to unmask] <[log in to unmask]> on behalf of Dejan Vitlacil <[log in to unmask]>
> Sent: 08 February 2023 09:24
> To: Oliver Freyermuth
> Cc: [log in to unmask]
> Subject: [External] Re: XRootD and tokens
>
> This email originated outside the University. Check before clicking links or attachments.


########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
--
--
Fábio Andrijauskas

--
--
Fábio Andrijauskas


Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1