Thanks Fabio,
Which version of XRootD are you running?
I’ll look deeper into your configuration and try to figure out if something is missing on my side.
Cheers,
Dejan
> On 8 Feb 2023, at 16:33, Fabio Andrijauskas <[log in to unmask]> wrote:
>
> Hi,
>
> I can see any obvious issue, this config is working for me:
>
>
> all.adminpath /tmp/xrootd/var/spool
> all.pidpath /tmp/xrootd/var/run
>
> oss.localroot /origin
> all.export /
>
> pfc.trace all
> ofs.trace all
> xrd.trace all -sched
> pss.setopt DebugLevel 5
> scitokens.trace all
>
>
> # Enable checksum
> xrootd.chksum adler32
>
> # Config TLS
> xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
> xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
> xrootd.tls capable all -data
>
> sec.level all compatible
> all.sitename ucsd
> # Enable Security
> xrootd.seclib libXrdSec.so
>
> # Enable "gsi" security
> #sec.protparm gsi -vomsfun:libXrdVoms.so -vomsfunparms:dbg
> #sec.protocol gsi -ca:1 -crl:3 -gridmap:/dev/null
>
> macaroons.secretkey /etc/xrootd/macaroon-secret
>
> # Authorizaton
> acc.audit deny
> acc.authdb /etc/xrootd/auth_file
> acc.authrefresh 60
> ofs.authorize 1
>
> # Xrootd TPC using rendezvous key
> ofs.tpc logok autorm pgm /etc/xrootd/xrdcp-tpc.sh
>
> # Env var needed by the above TPC script.
> setenv X509_USER_CERT = /etc/grid-security/xrd/xrdcert.pem
> setenv X509_USER_KEY = /etc/grid-security/xrd/xrdkey.pem
>
>
>
>
> ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
> ofs.authlib ++ libXrdMacaroons.so
>
> xrd.protocol http libXrdHttp.so
> http.exthandler xrdtpc libXrdHttpTPC.so
>
>
>
> cat /etc/xrootd/macaroon-secret
> SPKWus+p1S/dSpk15W9Cu/hCWeM0LnPuiNItzyAhkgUlUkAvzRYSOloI2HCSLKvk
> HzWOu3pTjlx1SsG2nyEyCw==
>
> cat /etc/xrootd/auth_file
> g /xrootd /data rl
>
>
>
> cat /etc/xrootd/scitokens.cfg
> [Global]
> onmissing = passthrough
>
> [Issuer OSG Monitoring]
> issuer = https://osg-htc.org/monitoring
> base_path = /
> map_subject = false
> default_user = xrootd
>
>
> cat /etc/xrootd/xrdcp-tpc.sh
> #!/bin/sh
> set -- `getopt S: -S 1 $*`
> while [ $# -gt 0 ]
> do
> case $1 in
> -S)
> ((nstreams=$2-1))
> [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams"
> shift 2
> ;;
> --)
> shift
> break
> ;;
> esac
> done
>
> src=$1
> dst=$2
> xrdcp --server $TCPstreamOpts -f $src root://$XRDXROOTD_PROXY/${dst}
>
> On Wed, Feb 8, 2023 at 5:34 AM Dejan Vitlacil <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>> Hi Fabio,
>>
>> Thanks for reaching out!
>>
>> This is command I’m using:
>> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo
>>
>> If I comment out “ofs.authorize” - there are no problems in uploading a file:
>> #############################################################################
>> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_NOauthz.repo
>> * We are completely uploaded and fine
>> * Closing connection 0
>> :-)
>> [centos@xrootd ~ ls -lh /data
>> total 16K
>> drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape
>> -rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt
>> -rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo
>> -rw-r--r--. 1 xrootd xrootd 168 Feb 8 13:17 testfile-token_NOauthz.repo
>> -rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo
>> [centos@xrootd ~]$
>> ##############################################################################
>>
>> I also added more tracing as you suggested.
>>
>> Cheers,
>> Dejan
>>
>> ###############################################################################
>> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo
>> * About to connect() to xrootd.e-commons.chalmers.se <http://xrootd.e-commons.chalmers.se/> port 80 (#0)
>> * Trying ::1...
>> * Connected to xrootd.e-commons.chalmers.se <http://xrootd.e-commons.chalmers.se/> (::1) port 80 (#0)
>> * Initializing NSS with certpath: sql:/etc/pki/nssdb
>> * CAfile: fullchain.pem
>> CApath: none
>> * NSS: client certificate not found (nickname not specified)
>> * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
>> * Server certificate:
>> * subject: CN=xrootd.e-commons.chalmers.se <http://xrootd.e-commons.chalmers.se/>
>> * start date: Jan 30 14:29:24 2023 GMT
>> * expire date: Apr 30 14:29:23 2023 GMT
>> * common name: xrootd.e-commons.chalmers.se <http://xrootd.e-commons.chalmers.se/>
>> * issuer: CN=R3,O=Let's Encrypt,C=US
>> > PUT /data/testfile-token_new.repo HTTP/1.1
>> > User-Agent: curl/7.29.0
>> > Host: xrootd.e-commons.chalmers.se <http://xrootd.e-commons.chalmers.se/>
>> > Accept: */*
>> > Authorization: Bearer eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjJkZWJhOWQxLTk1NTgtNDk2My05NWJjLTc1ZTk5M2UzYzgyZSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2FueSIsIm5iZiI6MTY3NTg1OTE2Mywic2NvcGUiOiJhZGRyZXNzIG9wZW5pZCBwcm9maWxlIHN0b3JhZ2Uuc3RhZ2U6XC8gZWR1cGVyc29uX2VudGl0bGVtZW50IHBob25lIG9mZmxpbmVfYWNjZXNzIGVkdXBlcnNvbl9zY29wZWRfYWZmaWxpYXRpb24gZWR1cGVyc29uX2Fzc3VyYW5jZSBlbWFpbCB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC9pYW0tZXNjYXBlLmNsb3VkLmNuYWYuaW5mbi5pdFwvIiwiZXhwIjoxNjc1ODYyNzYzLCJpYXQiOjE2NzU4NTkxNjMsImp0aSI6ImIyZTZjOGU0LTZlM2ItNDkzZS04ODcwLWNjNDY0NWYzMTk0OSIsImNsaWVudF9pZCI6ImMyNjU0ZWZiLWY4MTUtNGE2ZS04NDQ4LWIyZWU4MGVjYjE2YiIsIndsY2cuZ3JvdXBzIjpbIlwvZXNjYXBlIiwiXC9lc2NhcGVcL3NrYSJdfQ.L2DKM95W1ovF3QOIPrYR5ifGkyXlDgW2FwiKoSFm2XXAXVdzqrK36gQBCTu2hqoXaP9-6eU_a6Un0jXaY4Gi457HPUk4mDy8Mm0ZctaWAzOZnMyIIbvv0VKmEfFUDq_gBLMr1Lq2PbIuvHbhGhi58dyNlj4pdI8Ped19Q4fNXzg
>> > Content-Length: 168
>> > Expect: 100-continue
>> >
>> < HTTP/1.1 403 Forbidden
>> < Connection: Keep-Alive
>> < Server: XrootD/v5.5.1
>> < Content-Length: 66
>> * HTTP error before end of send, stop sending
>> <
>> Unable to create /data/testfile-token_new.repo; permission denied
>> * Closing connection 0
>> [centos@xrootd ~]$
>> #######################################################################
>>
>> [centos@xrootd ~]$ sudo cat /etc/xrootd/scitokens.cfg
>> [Global]
>> onmissing = passthrough
>> # don't use https://wlcg.cern.ch/jwt/v1/any on production instances
>> # audience = https://xrd.example.com:1094 <https://xrd.example.com:1094/>, https://wlcg.cern.ch/jwt/v1/any
>>
>> [Issuer ESCAPE IAM]
>> issuer = https://iam-escape.cloud.cnaf.infn.it/
>> base_path = /data
>> map_subject = false
>> default_user = xrootd
>> [centos@xrootd ~]$
>>
>> ########################################################################
>>
>> 230208 13:32:12 22541 Starting on Linux 3.10.0-1160.81.1.el7.x86_64
>> 230208 13:32:12 22541 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-http.cfg -k fifo -s /run/xrootd/xrootd-http.pid -n http
>> Copr. 2004-2012 Stanford University, xrd version v5.5.1
>> Config warning: this hostname, localhost, is registered without a domain qualification.
>> ++++++ xrootd http@localhost initialization started.
>> Config using configuration file /etc/xrootd/xrootd-http.cfg
>> =====> all.adminpath /var/spool/xrootd
>> =====> all.pidpath /run/xrootd
>> =====> xrd.protocol XrdHttp:80 libXrdHttp.so
>> =====> xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
>> =====> xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
>> =====> xrd.trace all -sched
>> =====> continue /etc/xrootd/config.d/
>> ++++++ xrootd http@localhost TLS initialization started.
>> ------ xrootd http@localhost TLS initialization ended.
>> Config maximum number of connections restricted to 65536
>> Config maximum number of threads restricted to 7149
>> 230208 13:32:12 22541 Xrd_Config: sendfile enabled.
>> 230208 13:32:12 22541 Xrd_LinkCtl: Allocating 64 link objects at a time
>> 230208 13:32:12 22541 Xrd_Poll: Starting poller 0
>> 230208 13:32:12 22541 Xrd_Poll: Starting poller 1
>> 230208 13:32:12 22541 Xrd_Poll: Starting poller 2
>> 230208 13:32:12 22541 Xrd_ProtLoad: protocol xroot wants to use port 1094
>> Plugin loaded xrdhttp v5.5.1 from protocol libXrdHttp-5.so
>> 230208 13:32:12 22541 Xrd_ProtLoad: protocol XrdHttp wants to use port 80
>> 230208 13:32:12 22541 Xrd_Config: xroot:1094 wsz=87380
>> 230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object xroot
>> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v5.5.1
>> ++++++ xroot protocol initialization started.
>> =====> all.export /data
>> =====> xrootd.tls capable all -data
>> =====> xrootd.seclib libXrdSec.so
>> =====> continue /etc/xrootd/config.d/
>> Config exporting /data
>> Plugin loaded secprot v5.5.1 from seclib libXrdSec-5.so
>> ++++++ Authentication system initialization started.
>> Plugin loaded secztn v5.5.1 from sec.protocol libXrdSecztn-5.so
>> =====> sec.protocol ztn
>> =====> continue /etc/xrootd/config.d/
>> Config 1 authentication directives processed in /etc/xrootd/xrootd-http.cfg
>> ------ Authentication system initialization completed.
>> ++++++ Protection system initialization started.
>> Config warning: Security level is set to none; request protection disabled!
>> Config Local protection level: none
>> Config Remote protection level: none
>> ------ Protection system initialization completed.
>> Config Routing for [::1]: local pub4 prv4 pub6 prv6
>> Config Route all4: 127.0.0.1 Dest=[::127.0.0.1]:1094
>> Config Route all6: [::1] Dest=[::1]:1094
>> ++++++ File system initialization started.
>> =====> ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
>> =====> ofs.trace all
>> =====> continue /etc/xrootd/config.d/
>> ++++++ Storage system initialization started.
>> =====> all.export /data
>> =====> continue /etc/xrootd/config.d/
>> Config effective /etc/xrootd/xrootd-http.cfg oss configuration:
>> oss.alloc 0 0 0
>> oss.spacescan 600
>> oss.fdlimit 32768 65536
>> oss.maxsize 0
>> oss.trace 0
>> oss.xfr 1 deny 10800 keep 1200
>> oss.memfile off max 963475456
>> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
>> oss.path /data r/w nocheck nodread nomig nopurge norcreate nostage
>> ------ Storage system initialization completed.
>> ++++++ Authorization system initialization started.
>> 230208 13:32:12 22541 acc_Config: Authorization system using configuration in /etc/xrootd/xrootd-http.cfg
>> =====> acc.authdb /etc/xrootd/Authfile
>> =====> continue /etc/xrootd/config.d/
>> Config 1 authorization directives processed in /etc/xrootd/xrootd-http.cfg
>> Config 1 auth entries processed in /etc/xrootd/Authfile
>> ------ Authorization system initialization completed.
>> Plugin loaded XrdAccSciTokens v5.5.1 from authlib libXrdAccSciTokens-5.so
>> ++++++ XrdAccSciTokens: Initialized SciTokens-based authorization.
>> =====> scitokens.trace all
>> =====> continue /etc/xrootd/config.d/
>> 230208 13:32:12 22541 scitokens_Config: Logging levels enabled - all
>> 230208 13:32:12 22541 scitokens_Reconfig: Parsing configuration file: /etc/xrootd/scitokens.cfg
>> 230208 13:32:12 22541 scitokens_Reconfig: Configuring issuer https://iam-escape.cloud.cnaf.infn.it/
>> ++++++ Checkpoint initialization started.
>> ++++++ Checkpoint initialization completed.
>> Config effective /etc/xrootd/xrootd-http.cfg ofs configuration:
>> all.role server
>> ofs.authorize
>> ofs.maxdelay 60
>> ofs.persist manual hold 600 logdir /var/spool/xrootd/http/.ofs/posc.log
>> ofs.trace ffff
>> ofs.authlib default
>> ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
>> ------ File system server initialization completed.
>> Config asynchronous I/O has been disabled!
>> 230208 13:32:12 22541 ofs_FAttr: FAttr req=info
>> ------ xroot protocol initialization completed.
>> 230208 13:32:12 22541 Xrd_ProtLoad: enabling port 1094 for protocol xroot
>> 230208 13:32:12 22541 Xrd_Config: XrdHttp:80 wsz=87380
>> 230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object XrdHttp
>> Copr. 2012 CERN IT, an HTTP implementation for the XRootD framework.
>> ++++++ HTTP protocol initialization started.
>> =====> http.header2cgi Authorization authz
>> =====> continue /etc/xrootd/config.d/
>> Config Using xrd.tls to supply 'cert' and 'key'.
>> Config Using xrd.tlsca to supply 'cadir'.
>> ++++++ HTTPS initialization started.
>> ------ HTTPS initialization completed.
>> 230208 13:32:12 22541 sysConfig: XRDROLE: server
>> 230208 13:32:12 22541 sysConfig: Configured as HTTP(s) data server.
>> ------ HTTP protocol initialization completed.
>> 230208 13:32:12 22541 Xrd_ProtLoad: enabling port 80 for protocol XrdHttp
>> ------ xrootd http@localhost:80 initialization completed.
>> 230208 13:32:12 22558 TLS_Refresh: CRL refresh started.
>> 230208 13:32:12 22558 TLS_Refresh: CRL refresh will happen in 28800 seconds.
>> 230208 13:32:12 22550 TLS_Refresh: CRL refresh started.
>> 230208 13:32:12 22550 TLS_Refresh: CRL refresh will happen in 28800 seconds.
>> 230208 13:32:32 22546 Xrd_Inet: Accepted connection on port 80 from 25@localhost
>> 230208 13:32:32 22546 Xrd_ProtLoad: matched port 80 protocol XrdHttp
>> 230208 13:32:32 22546 anon.0:25@localhost Xrd_Poll: FD 25 attached to poller 0; num=1
>> 230208 13:32:32 22546 XrootdBridge: unknown.1:25@localhost login as nobody
>> 230208 13:32:32 22546 unknown.1:25@localhost ofs_open: 200-40664 fn=/data/testfile-token_WITHauthz.repo
>> 230208 13:32:32 22546 scitokens_Access: Trying token-based access control
>> 230208 13:32:32 22546 scitokens_Access: Token not found in recent cache; parsing.
>> 230208 13:32:32 22546 scitokens_Access: New valid token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
>> 230208 13:32:32 22546 scitokens_Access: Trying token-based access control
>> 230208 13:32:32 22546 scitokens_Access: Cached token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
>> 230208 13:32:32 22546 ofs_open: unknown.1:25@localhost Unable to create /data/testfile-token_WITHauthz.repo; permission denied
>> 230208 13:32:32 22546 unknown.1:25@localhost ofs_close: use=0 fn=dummy
>> 230208 13:32:32 22546 XrootdXeq: unknown.1:25@localhost disc 0:00:00 (send failure)
>> 230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: Poller 0 removing FD 25
>> 230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: FD 25 detached from poller 0; num=0
>> [centos@xrootd ~]$
>>
>>
>>> On 8 Feb 2023, at 12:42, Fabio Andrijauskas <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>>>
>>> Hi,
>>>
>>> Can you send how are you requesting/creating the files?
>>>
>>> Try to add this to get all the possible logs:
>>>
>>> pfc.trace all
>>> ofs.trace all
>>> xrd.trace all -sched
>>> pss.setopt DebugLevel 5
>>> scitokens.trace all
>>>
>>> Did you try to create the the path on the export path?
>>>
>>> Can you send your sci token config file?
>>>
>>>
>>>
>>> On Wed, Feb 8, 2023 at 2:25 AM Dejan Vitlacil <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>>>> Hi Matt,
>>>>
>>>> I run out of ideas, so all suggestions are appreciated.
>>>> I think unix bits are in place.
>>>>
>>>> Cheers,
>>>> Dejan
>>>>
>>>> ……
>>>> [centos@xrootd ~]$ ls -lh / |grep data
>>>> drwxr-xr-x. 3 xrootd xrootd 84 Feb 6 15:22 data
>>>> [centos@xrootd ~]$ ls -lh /data
>>>> total 12K
>>>> drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape
>>>> -rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt
>>>> -rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo
>>>> -rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo
>>>> [centos@xrootd ~]$
>>>>
>>>>
>>>> > On 8 Feb 2023, at 11:05, Doidge, Matt <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>>>> >
>>>> > Hello,
>>>> > This is a bit of a low-level suggestion, but can the xrootd unix user write to /data? I had similar looking issues with a test xrootd server and a known working config where I had forgotten to chown the exported path.
>>>> >
>>>> > My apologies for the noise if you've already checked this.
>>>> >
>>>> > Cheers,
>>>> > Matt
>>>> >
>>>> > ________________________________________
>>>> > From: [log in to unmask] <mailto:[log in to unmask]> <[log in to unmask] <mailto:[log in to unmask]>> on behalf of Dejan Vitlacil <[log in to unmask] <mailto:[log in to unmask]>>
>>>> > Sent: 08 February 2023 09:24
>>>> > To: Oliver Freyermuth
>>>> > Cc: [log in to unmask] <mailto:[log in to unmask]>
>>>> > Subject: [External] Re: XRootD and tokens
>>>> >
>>>> > This email originated outside the University. Check before clicking links or attachments.
>>>>
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>> --
>>> --
>>> Fábio Andrijauskas
>>
> --
> --
> Fábio Andrijauskas
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
