Hi Fabio,
Thanks for reaching out!
This is command I’m using:
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo
If I comment out “ofs.authorize” - there are no problems in uploading a file:
#############################################################################
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_NOauthz.repo
* We are completely uploaded and fine
* Closing connection 0
:-)
[centos@xrootd ~ ls -lh /data
total 16K
drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape
-rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt
-rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo
-rw-r--r--. 1 xrootd xrootd 168 Feb 8 13:17 testfile-token_NOauthz.repo
-rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo
[centos@xrootd ~]$
##############################################################################
I also added more tracing as you suggested.
Cheers,
Dejan
###############################################################################
[centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H "Authorization: Bearer $AT" --cacert fullchain.pem https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo
* About to connect() to xrootd.e-commons.chalmers.se port 80 (#0)
* Trying ::1...
* Connected to xrootd.e-commons.chalmers.se (::1) port 80 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: fullchain.pem
CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=xrootd.e-commons.chalmers.se
* start date: Jan 30 14:29:24 2023 GMT
* expire date: Apr 30 14:29:23 2023 GMT
* common name: xrootd.e-commons.chalmers.se
* issuer: CN=R3,O=Let's Encrypt,C=US
> PUT /data/testfile-token_new.repo HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xrootd.e-commons.chalmers.se
> Accept: */*
> Authorization: Bearer eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjJkZWJhOWQxLTk1NTgtNDk2My05NWJjLTc1ZTk5M2UzYzgyZSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2FueSIsIm5iZiI6MTY3NTg1OTE2Mywic2NvcGUiOiJhZGRyZXNzIG9wZW5pZCBwcm9maWxlIHN0b3JhZ2Uuc3RhZ2U6XC8gZWR1cGVyc29uX2VudGl0bGVtZW50IHBob25lIG9mZmxpbmVfYWNjZXNzIGVkdXBlcnNvbl9zY29wZWRfYWZmaWxpYXRpb24gZWR1cGVyc29uX2Fzc3VyYW5jZSBlbWFpbCB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC9pYW0tZXNjYXBlLmNsb3VkLmNuYWYuaW5mbi5pdFwvIiwiZXhwIjoxNjc1ODYyNzYzLCJpYXQiOjE2NzU4NTkxNjMsImp0aSI6ImIyZTZjOGU0LTZlM2ItNDkzZS04ODcwLWNjNDY0NWYzMTk0OSIsImNsaWVudF9pZCI6ImMyNjU0ZWZiLWY4MTUtNGE2ZS04NDQ4LWIyZWU4MGVjYjE2YiIsIndsY2cuZ3JvdXBzIjpbIlwvZXNjYXBlIiwiXC9lc2NhcGVcL3NrYSJdfQ.L2DKM95W1ovF3QOIPrYR5ifGkyXlDgW2FwiKoSFm2XXAXVdzqrK36gQBCTu2hqoXaP9-6eU_a6Un0jXaY4Gi457HPUk4mDy8Mm0ZctaWAzOZnMyIIbvv0VKmEfFUDq_gBLMr1Lq2PbIuvHbhGhi58dyNlj4pdI8Ped19Q4fNXzg
> Content-Length: 168
> Expect: 100-continue
>
< HTTP/1.1 403 Forbidden
< Connection: Keep-Alive
< Server: XrootD/v5.5.1
< Content-Length: 66
* HTTP error before end of send, stop sending
<
Unable to create /data/testfile-token_new.repo; permission denied
* Closing connection 0
[centos@xrootd ~]$
#######################################################################
[centos@xrootd ~]$ sudo cat /etc/xrootd/scitokens.cfg
[Global]
onmissing = passthrough
# don't use https://wlcg.cern.ch/jwt/v1/any on production instances
# audience = https://xrd.example.com:1094, https://wlcg.cern.ch/jwt/v1/any
[Issuer ESCAPE IAM]
issuer = https://iam-escape.cloud.cnaf.infn.it/
base_path = /data
map_subject = false
default_user = xrootd
[centos@xrootd ~]$
########################################################################
230208 13:32:12 22541 Starting on Linux 3.10.0-1160.81.1.el7.x86_64
230208 13:32:12 22541 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-http.cfg -k fifo -s /run/xrootd/xrootd-http.pid -n http
Copr. 2004-2012 Stanford University, xrd version v5.5.1
Config warning: this hostname, localhost, is registered without a domain qualification.
++++++ xrootd http@localhost initialization started.
Config using configuration file /etc/xrootd/xrootd-http.cfg
=====> all.adminpath /var/spool/xrootd
=====> all.pidpath /run/xrootd
=====> xrd.protocol XrdHttp:80 libXrdHttp.so
=====> xrd.tls /etc/grid-security/xrd/xrdcert.pem /etc/grid-security/xrd/xrdkey.pem
=====> xrd.tlsca certdir /etc/grid-security/certificates refresh 8h
=====> xrd.trace all -sched
=====> continue /etc/xrootd/config.d/
++++++ xrootd http@localhost TLS initialization started.
------ xrootd http@localhost TLS initialization ended.
Config maximum number of connections restricted to 65536
Config maximum number of threads restricted to 7149
230208 13:32:12 22541 Xrd_Config: sendfile enabled.
230208 13:32:12 22541 Xrd_LinkCtl: Allocating 64 link objects at a time
230208 13:32:12 22541 Xrd_Poll: Starting poller 0
230208 13:32:12 22541 Xrd_Poll: Starting poller 1
230208 13:32:12 22541 Xrd_Poll: Starting poller 2
230208 13:32:12 22541 Xrd_ProtLoad: protocol xroot wants to use port 1094
Plugin loaded xrdhttp v5.5.1 from protocol libXrdHttp-5.so
230208 13:32:12 22541 Xrd_ProtLoad: protocol XrdHttp wants to use port 80
230208 13:32:12 22541 Xrd_Config: xroot:1094 wsz=87380
230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object xroot
Copr. 2012 Stanford University, xroot protocol 5.1.0 version v5.5.1
++++++ xroot protocol initialization started.
=====> all.export /data
=====> xrootd.tls capable all -data
=====> xrootd.seclib libXrdSec.so
=====> continue /etc/xrootd/config.d/
Config exporting /data
Plugin loaded secprot v5.5.1 from seclib libXrdSec-5.so
++++++ Authentication system initialization started.
Plugin loaded secztn v5.5.1 from sec.protocol libXrdSecztn-5.so
=====> sec.protocol ztn
=====> continue /etc/xrootd/config.d/
Config 1 authentication directives processed in /etc/xrootd/xrootd-http.cfg
------ Authentication system initialization completed.
++++++ Protection system initialization started.
Config warning: Security level is set to none; request protection disabled!
Config Local protection level: none
Config Remote protection level: none
------ Protection system initialization completed.
Config Routing for [::1]: local pub4 prv4 pub6 prv6
Config Route all4: 127.0.0.1 Dest=[::127.0.0.1]:1094
Config Route all6: [::1] Dest=[::1]:1094
++++++ File system initialization started.
=====> ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
=====> ofs.trace all
=====> continue /etc/xrootd/config.d/
++++++ Storage system initialization started.
=====> all.export /data
=====> continue /etc/xrootd/config.d/
Config effective /etc/xrootd/xrootd-http.cfg oss configuration:
oss.alloc 0 0 0
oss.spacescan 600
oss.fdlimit 32768 65536
oss.maxsize 0
oss.trace 0
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 963475456
oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage
oss.path /data r/w nocheck nodread nomig nopurge norcreate nostage
------ Storage system initialization completed.
++++++ Authorization system initialization started.
230208 13:32:12 22541 acc_Config: Authorization system using configuration in /etc/xrootd/xrootd-http.cfg
=====> acc.authdb /etc/xrootd/Authfile
=====> continue /etc/xrootd/config.d/
Config 1 authorization directives processed in /etc/xrootd/xrootd-http.cfg
Config 1 auth entries processed in /etc/xrootd/Authfile
------ Authorization system initialization completed.
Plugin loaded XrdAccSciTokens v5.5.1 from authlib libXrdAccSciTokens-5.so
++++++ XrdAccSciTokens: Initialized SciTokens-based authorization.
=====> scitokens.trace all
=====> continue /etc/xrootd/config.d/
230208 13:32:12 22541 scitokens_Config: Logging levels enabled - all
230208 13:32:12 22541 scitokens_Reconfig: Parsing configuration file: /etc/xrootd/scitokens.cfg
230208 13:32:12 22541 scitokens_Reconfig: Configuring issuer https://iam-escape.cloud.cnaf.infn.it/
++++++ Checkpoint initialization started.
++++++ Checkpoint initialization completed.
Config effective /etc/xrootd/xrootd-http.cfg ofs configuration:
all.role server
ofs.authorize
ofs.maxdelay 60
ofs.persist manual hold 600 logdir /var/spool/xrootd/http/.ofs/posc.log
ofs.trace ffff
ofs.authlib default
ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg
------ File system server initialization completed.
Config asynchronous I/O has been disabled!
230208 13:32:12 22541 ofs_FAttr: FAttr req=info
------ xroot protocol initialization completed.
230208 13:32:12 22541 Xrd_ProtLoad: enabling port 1094 for protocol xroot
230208 13:32:12 22541 Xrd_Config: XrdHttp:80 wsz=87380
230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object XrdHttp
Copr. 2012 CERN IT, an HTTP implementation for the XRootD framework.
++++++ HTTP protocol initialization started.
=====> http.header2cgi Authorization authz
=====> continue /etc/xrootd/config.d/
Config Using xrd.tls to supply 'cert' and 'key'.
Config Using xrd.tlsca to supply 'cadir'.
++++++ HTTPS initialization started.
------ HTTPS initialization completed.
230208 13:32:12 22541 sysConfig: XRDROLE: server
230208 13:32:12 22541 sysConfig: Configured as HTTP(s) data server.
------ HTTP protocol initialization completed.
230208 13:32:12 22541 Xrd_ProtLoad: enabling port 80 for protocol XrdHttp
------ xrootd http@localhost:80 initialization completed.
230208 13:32:12 22558 TLS_Refresh: CRL refresh started.
230208 13:32:12 22558 TLS_Refresh: CRL refresh will happen in 28800 seconds.
230208 13:32:12 22550 TLS_Refresh: CRL refresh started.
230208 13:32:12 22550 TLS_Refresh: CRL refresh will happen in 28800 seconds.
230208 13:32:32 22546 Xrd_Inet: Accepted connection on port 80 from 25@localhost
230208 13:32:32 22546 Xrd_ProtLoad: matched port 80 protocol XrdHttp
230208 13:32:32 22546 anon.0:25@localhost Xrd_Poll: FD 25 attached to poller 0; num=1
230208 13:32:32 22546 XrootdBridge: unknown.1:25@localhost login as nobody
230208 13:32:32 22546 unknown.1:25@localhost ofs_open: 200-40664 fn=/data/testfile-token_WITHauthz.repo
230208 13:32:32 22546 scitokens_Access: Trying token-based access control
230208 13:32:32 22546 scitokens_Access: Token not found in recent cache; parsing.
230208 13:32:32 22546 scitokens_Access: New valid token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
230208 13:32:32 22546 scitokens_Access: Trying token-based access control
230208 13:32:32 22546 scitokens_Access: Cached token mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska
230208 13:32:32 22546 ofs_open: unknown.1:25@localhost Unable to create /data/testfile-token_WITHauthz.repo; permission denied
230208 13:32:32 22546 unknown.1:25@localhost ofs_close: use=0 fn=dummy
230208 13:32:32 22546 XrootdXeq: unknown.1:25@localhost disc 0:00:00 (send failure)
230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: Poller 0 removing FD 25
230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: FD 25 detached from poller 0; num=0
[centos@xrootd ~]$
> On 8 Feb 2023, at 12:42, Fabio Andrijauskas <[log in to unmask]> wrote:
>
> Hi,
>
> Can you send how are you requesting/creating the files?
>
> Try to add this to get all the possible logs:
>
> pfc.trace all
> ofs.trace all
> xrd.trace all -sched
> pss.setopt DebugLevel 5
> scitokens.trace all
>
> Did you try to create the the path on the export path?
>
> Can you send your sci token config file?
>
>
>
> On Wed, Feb 8, 2023 at 2:25 AM Dejan Vitlacil <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>> Hi Matt,
>>
>> I run out of ideas, so all suggestions are appreciated.
>> I think unix bits are in place.
>>
>> Cheers,
>> Dejan
>>
>> ……
>> [centos@xrootd ~]$ ls -lh / |grep data
>> drwxr-xr-x. 3 xrootd xrootd 84 Feb 6 15:22 data
>> [centos@xrootd ~]$ ls -lh /data
>> total 12K
>> drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape
>> -rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt
>> -rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo
>> -rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo
>> [centos@xrootd ~]$
>>
>>
>> > On 8 Feb 2023, at 11:05, Doidge, Matt <[log in to unmask] <mailto:[log in to unmask]>> wrote:
>> >
>> > Hello,
>> > This is a bit of a low-level suggestion, but can the xrootd unix user write to /data? I had similar looking issues with a test xrootd server and a known working config where I had forgotten to chown the exported path.
>> >
>> > My apologies for the noise if you've already checked this.
>> >
>> > Cheers,
>> > Matt
>> >
>> > ________________________________________
>> > From: [log in to unmask] <mailto:[log in to unmask]> <[log in to unmask] <mailto:[log in to unmask]>> on behalf of Dejan Vitlacil <[log in to unmask] <mailto:[log in to unmask]>>
>> > Sent: 08 February 2023 09:24
>> > To: Oliver Freyermuth
>> > Cc: [log in to unmask] <mailto:[log in to unmask]>
>> > Subject: [External] Re: XRootD and tokens
>> >
>> > This email originated outside the University. Check before clicking links or attachments.
>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> --
> --
> Fábio Andrijauskas
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
