Hi Dejan, I am using the v5.5.1 on docker image opensciencegrid/xrootd-standalone:3.6-release-20230205-0812 *--Fábio Andrijauskas* On Thu, Feb 9, 2023 at 11:42 PM Dejan Vitlacil <[log in to unmask]> wrote: > Thanks Fabio, > > Which version of XRootD are you running? > I’ll look deeper into your configuration and try to figure out if > something is missing on my side. > > Cheers, > Dejan > > > On 8 Feb 2023, at 16:33, Fabio Andrijauskas <[log in to unmask]> > wrote: > > Hi, > > I can see any obvious issue, this config is working for me: > > > all.adminpath /tmp/xrootd/var/spool > all.pidpath /tmp/xrootd/var/run > > > oss.localroot /origin > all.export / > > > pfc.trace all > ofs.trace all > xrd.trace all -sched > pss.setopt DebugLevel 5 > scitokens.trace all > > > > > # Enable checksum > xrootd.chksum adler32 > > > # Config TLS > xrd.tls /etc/grid-security/xrd/xrdcert.pem > /etc/grid-security/xrd/xrdkey.pem > xrd.tlsca certdir /etc/grid-security/certificates refresh 8h > xrootd.tls capable all -data > > > sec.level all compatible > all.sitename ucsd > # Enable Security > xrootd.seclib libXrdSec.so > > > # Enable "gsi" security > #sec.protparm gsi -vomsfun:libXrdVoms.so -vomsfunparms:dbg > #sec.protocol gsi -ca:1 -crl:3 -gridmap:/dev/null > > > macaroons.secretkey /etc/xrootd/macaroon-secret > > > # Authorizaton > acc.audit deny > acc.authdb /etc/xrootd/auth_file > acc.authrefresh 60 > ofs.authorize 1 > > > # Xrootd TPC using rendezvous key > ofs.tpc logok autorm pgm /etc/xrootd/xrdcp-tpc.sh > > > # Env var needed by the above TPC script. > setenv X509_USER_CERT = /etc/grid-security/xrd/xrdcert.pem > setenv X509_USER_KEY = /etc/grid-security/xrd/xrdkey.pem > > > > > > > > > ofs.authlib ++ libXrdAccSciTokens.so config=/etc/xrootd/scitokens.cfg > ofs.authlib ++ libXrdMacaroons.so > > > xrd.protocol http libXrdHttp.so > http.exthandler xrdtpc libXrdHttpTPC.so > > > > > > > cat /etc/xrootd/macaroon-secret > SPKWus+p1S/dSpk15W9Cu/hCWeM0LnPuiNItzyAhkgUlUkAvzRYSOloI2HCSLKvk > HzWOu3pTjlx1SsG2nyEyCw== > > > cat /etc/xrootd/auth_file > g /xrootd /data rl > > > > > > > cat /etc/xrootd/scitokens.cfg > [Global] > onmissing = passthrough > > > [Issuer OSG Monitoring] > issuer = https://osg-htc.org/monitoring > base_path = / > map_subject = false > default_user = xrootd > > > cat /etc/xrootd/xrdcp-tpc.sh > #!/bin/sh > set -- `getopt S: -S 1 $*` > while [ $# -gt 0 ] > do > case $1 in > -S) > ((nstreams=$2-1)) > [ $nstreams -ge 1 ] && TCPstreamOpts="-S $nstreams" > shift 2 > ;; > --) > shift > break > ;; > esac > done > > > src=$1 > dst=$2 > xrdcp --server $TCPstreamOpts -f $src root://$XRDXROOTD_PROXY/${dst} > > On Wed, Feb 8, 2023 at 5:34 AM Dejan Vitlacil <[log in to unmask]> > wrote: > >> Hi Fabio, >> >> Thanks for reaching out! >> >> This is command I’m using: >> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H >> "Authorization: Bearer $AT" --cacert fullchain.pem >> https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo >> >> If I comment out “ofs.authorize” - there are no problems in uploading a >> file: >> >> ############################################################################# >> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H >> "Authorization: Bearer $AT" --cacert fullchain.pem >> https://xrootd.e-commons.chalmers.se:80/data/testfile-token_NOauthz.repo >> * We are completely uploaded and fine >> * Closing connection 0 >> :-) >> [centos@xrootd ~ ls -lh /data >> total 16K >> drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape >> -rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt >> -rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo >> -rw-r--r--. 1 xrootd xrootd 168 Feb 8 13:17 testfile-token_NOauthz.repo >> -rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo >> [centos@xrootd ~]$ >> >> ############################################################################## >> >> I also added more tracing as you suggested. >> >> Cheers, >> Dejan >> >> >> ############################################################################### >> [centos@xrootd ~]$ curl -v -X PUT --upload-file test.repo -H >> "Authorization: Bearer $AT" --cacert fullchain.pem >> https://xrootd.e-commons.chalmers.se:80/data/testfile-token_new.repo >> * About to connect() to xrootd.e-commons.chalmers.se port 80 (#0) >> * Trying ::1... >> * Connected to xrootd.e-commons.chalmers.se (::1) port 80 (#0) >> * Initializing NSS with certpath: sql:/etc/pki/nssdb >> * CAfile: fullchain.pem >> CApath: none >> * NSS: client certificate not found (nickname not specified) >> * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 >> * Server certificate: >> * subject: CN=xrootd.e-commons.chalmers.se >> * start date: Jan 30 14:29:24 2023 GMT >> * expire date: Apr 30 14:29:23 2023 GMT >> * common name: xrootd.e-commons.chalmers.se >> * issuer: CN=R3,O=Let's Encrypt,C=US >> > PUT /data/testfile-token_new.repo HTTP/1.1 >> > User-Agent: curl/7.29.0 >> > Host: xrootd.e-commons.chalmers.se >> > Accept: */* >> > Authorization: Bearer >> eyJraWQiOiJyc2ExIiwiYWxnIjoiUlMyNTYifQ.eyJ3bGNnLnZlciI6IjEuMCIsInN1YiI6IjJkZWJhOWQxLTk1NTgtNDk2My05NWJjLTc1ZTk5M2UzYzgyZSIsImF1ZCI6Imh0dHBzOlwvXC93bGNnLmNlcm4uY2hcL2p3dFwvdjFcL2FueSIsIm5iZiI6MTY3NTg1OTE2Mywic2NvcGUiOiJhZGRyZXNzIG9wZW5pZCBwcm9maWxlIHN0b3JhZ2Uuc3RhZ2U6XC8gZWR1cGVyc29uX2VudGl0bGVtZW50IHBob25lIG9mZmxpbmVfYWNjZXNzIGVkdXBlcnNvbl9zY29wZWRfYWZmaWxpYXRpb24gZWR1cGVyc29uX2Fzc3VyYW5jZSBlbWFpbCB3bGNnLmdyb3VwcyIsImlzcyI6Imh0dHBzOlwvXC9pYW0tZXNjYXBlLmNsb3VkLmNuYWYuaW5mbi5pdFwvIiwiZXhwIjoxNjc1ODYyNzYzLCJpYXQiOjE2NzU4NTkxNjMsImp0aSI6ImIyZTZjOGU0LTZlM2ItNDkzZS04ODcwLWNjNDY0NWYzMTk0OSIsImNsaWVudF9pZCI6ImMyNjU0ZWZiLWY4MTUtNGE2ZS04NDQ4LWIyZWU4MGVjYjE2YiIsIndsY2cuZ3JvdXBzIjpbIlwvZXNjYXBlIiwiXC9lc2NhcGVcL3NrYSJdfQ.L2DKM95W1ovF3QOIPrYR5ifGkyXlDgW2FwiKoSFm2XXAXVdzqrK36gQBCTu2hqoXaP9-6eU_a6Un0jXaY4Gi457HPUk4mDy8Mm0ZctaWAzOZnMyIIbvv0VKmEfFUDq_gBLMr1Lq2PbIuvHbhGhi58dyNlj4pdI8Ped19Q4fNXzg >> > Content-Length: 168 >> > Expect: 100-continue >> > >> < HTTP/1.1 403 Forbidden >> < Connection: Keep-Alive >> < Server: XrootD/v5.5.1 >> < Content-Length: 66 >> * HTTP error before end of send, stop sending >> < >> Unable to create /data/testfile-token_new.repo; permission denied >> * Closing connection 0 >> [centos@xrootd ~]$ >> ####################################################################### >> >> [centos@xrootd ~]$ sudo cat /etc/xrootd/scitokens.cfg >> [Global] >> onmissing = passthrough >> # don't use https://wlcg.cern.ch/jwt/v1/any on production instances >> # audience = https://xrd.example.com:1094, >> https://wlcg.cern.ch/jwt/v1/any >> >> [Issuer ESCAPE IAM] >> issuer = https://iam-escape.cloud.cnaf.infn.it/ >> base_path = /data >> map_subject = false >> default_user = xrootd >> [centos@xrootd ~]$ >> >> ######################################################################## >> >> 230208 13:32:12 22541 Starting on Linux 3.10.0-1160.81.1.el7.x86_64 >> 230208 13:32:12 22541 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c >> /etc/xrootd/xrootd-http.cfg -k fifo -s /run/xrootd/xrootd-http.pid -n http >> Copr. 2004-2012 Stanford University, xrd version v5.5.1 >> Config warning: this hostname, localhost, is registered without a domain >> qualification. >> ++++++ xrootd http@localhost initialization started. >> Config using configuration file /etc/xrootd/xrootd-http.cfg >> =====> all.adminpath /var/spool/xrootd >> =====> all.pidpath /run/xrootd >> =====> xrd.protocol XrdHttp:80 libXrdHttp.so >> =====> xrd.tls /etc/grid-security/xrd/xrdcert.pem >> /etc/grid-security/xrd/xrdkey.pem >> =====> xrd.tlsca certdir /etc/grid-security/certificates refresh 8h >> =====> xrd.trace all -sched >> =====> continue /etc/xrootd/config.d/ >> ++++++ xrootd http@localhost TLS initialization started. >> ------ xrootd http@localhost TLS initialization ended. >> Config maximum number of connections restricted to 65536 >> Config maximum number of threads restricted to 7149 >> 230208 13:32:12 22541 Xrd_Config: sendfile enabled. >> 230208 13:32:12 22541 Xrd_LinkCtl: Allocating 64 link objects at a time >> 230208 13:32:12 22541 Xrd_Poll: Starting poller 0 >> 230208 13:32:12 22541 Xrd_Poll: Starting poller 1 >> 230208 13:32:12 22541 Xrd_Poll: Starting poller 2 >> 230208 13:32:12 22541 Xrd_ProtLoad: protocol xroot wants to use port 1094 >> Plugin loaded xrdhttp v5.5.1 from protocol libXrdHttp-5.so >> 230208 13:32:12 22541 Xrd_ProtLoad: protocol XrdHttp wants to use port 80 >> 230208 13:32:12 22541 Xrd_Config: xroot:1094 wsz=87380 >> 230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object xroot >> Copr. 2012 Stanford University, xroot protocol 5.1.0 version v5.5.1 >> ++++++ xroot protocol initialization started. >> =====> all.export /data >> =====> xrootd.tls capable all -data >> =====> xrootd.seclib libXrdSec.so >> =====> continue /etc/xrootd/config.d/ >> Config exporting /data >> Plugin loaded secprot v5.5.1 from seclib libXrdSec-5.so >> ++++++ Authentication system initialization started. >> Plugin loaded secztn v5.5.1 from sec.protocol libXrdSecztn-5.so >> =====> sec.protocol ztn >> =====> continue /etc/xrootd/config.d/ >> Config 1 authentication directives processed in >> /etc/xrootd/xrootd-http.cfg >> ------ Authentication system initialization completed. >> ++++++ Protection system initialization started. >> Config warning: Security level is set to none; request protection >> disabled! >> Config Local protection level: none >> Config Remote protection level: none >> ------ Protection system initialization completed. >> Config Routing for [::1]: local pub4 prv4 pub6 prv6 >> Config Route all4: 127.0.0.1 Dest=[::127.0.0.1]:1094 >> Config Route all6: [::1] Dest=[::1]:1094 >> ++++++ File system initialization started. >> =====> ofs.authlib ++ libXrdAccSciTokens.so >> config=/etc/xrootd/scitokens.cfg >> =====> ofs.trace all >> =====> continue /etc/xrootd/config.d/ >> ++++++ Storage system initialization started. >> =====> all.export /data >> =====> continue /etc/xrootd/config.d/ >> Config effective /etc/xrootd/xrootd-http.cfg oss configuration: >> oss.alloc 0 0 0 >> oss.spacescan 600 >> oss.fdlimit 32768 65536 >> oss.maxsize 0 >> oss.trace 0 >> oss.xfr 1 deny 10800 keep 1200 >> oss.memfile off max 963475456 >> oss.defaults r/w nocheck nodread nomig nopurge norcreate nostage >> oss.path /data r/w nocheck nodread nomig nopurge norcreate nostage >> ------ Storage system initialization completed. >> ++++++ Authorization system initialization started. >> 230208 13:32:12 22541 acc_Config: Authorization system using >> configuration in /etc/xrootd/xrootd-http.cfg >> =====> acc.authdb /etc/xrootd/Authfile >> =====> continue /etc/xrootd/config.d/ >> Config 1 authorization directives processed in /etc/xrootd/xrootd-http.cfg >> Config 1 auth entries processed in /etc/xrootd/Authfile >> ------ Authorization system initialization completed. >> Plugin loaded XrdAccSciTokens v5.5.1 from authlib libXrdAccSciTokens-5.so >> ++++++ XrdAccSciTokens: Initialized SciTokens-based authorization. >> =====> scitokens.trace all >> =====> continue /etc/xrootd/config.d/ >> 230208 13:32:12 22541 scitokens_Config: Logging levels enabled - all >> 230208 13:32:12 22541 scitokens_Reconfig: Parsing configuration file: >> /etc/xrootd/scitokens.cfg >> 230208 13:32:12 22541 scitokens_Reconfig: Configuring issuer >> https://iam-escape.cloud.cnaf.infn.it/ >> ++++++ Checkpoint initialization started. >> ++++++ Checkpoint initialization completed. >> Config effective /etc/xrootd/xrootd-http.cfg ofs configuration: >> all.role server >> ofs.authorize >> ofs.maxdelay 60 >> ofs.persist manual hold 600 logdir >> /var/spool/xrootd/http/.ofs/posc.log >> ofs.trace ffff >> ofs.authlib default >> ofs.authlib ++ libXrdAccSciTokens.so >> config=/etc/xrootd/scitokens.cfg >> ------ File system server initialization completed. >> Config asynchronous I/O has been disabled! >> 230208 13:32:12 22541 ofs_FAttr: FAttr req=info >> ------ xroot protocol initialization completed. >> 230208 13:32:12 22541 Xrd_ProtLoad: enabling port 1094 for protocol xroot >> 230208 13:32:12 22541 Xrd_Config: XrdHttp:80 wsz=87380 >> 230208 13:32:12 22541 Xrd_ProtLoad: getting protocol object XrdHttp >> Copr. 2012 CERN IT, an HTTP implementation for the XRootD framework. >> ++++++ HTTP protocol initialization started. >> =====> http.header2cgi Authorization authz >> =====> continue /etc/xrootd/config.d/ >> Config Using xrd.tls to supply 'cert' and 'key'. >> Config Using xrd.tlsca to supply 'cadir'. >> ++++++ HTTPS initialization started. >> ------ HTTPS initialization completed. >> 230208 13:32:12 22541 sysConfig: XRDROLE: server >> 230208 13:32:12 22541 sysConfig: Configured as HTTP(s) data server. >> ------ HTTP protocol initialization completed. >> 230208 13:32:12 22541 Xrd_ProtLoad: enabling port 80 for protocol XrdHttp >> ------ xrootd http@localhost:80 initialization completed. >> 230208 13:32:12 22558 TLS_Refresh: CRL refresh started. >> 230208 13:32:12 22558 TLS_Refresh: CRL refresh will happen in 28800 >> seconds. >> 230208 13:32:12 22550 TLS_Refresh: CRL refresh started. >> 230208 13:32:12 22550 TLS_Refresh: CRL refresh will happen in 28800 >> seconds. >> 230208 13:32:32 22546 Xrd_Inet: Accepted connection on port 80 from >> 25@localhost >> 230208 13:32:32 22546 Xrd_ProtLoad: matched port 80 protocol XrdHttp >> 230208 13:32:32 22546 anon.0:25@localhost Xrd_Poll: FD 25 attached to >> poller 0; num=1 >> 230208 13:32:32 22546 XrootdBridge: unknown.1:25@localhost login as >> nobody >> 230208 13:32:32 22546 unknown.1:25@localhost ofs_open: 200-40664 >> fn=/data/testfile-token_WITHauthz.repo >> 230208 13:32:32 22546 scitokens_Access: Trying token-based access control >> 230208 13:32:32 22546 scitokens_Access: Token not found in recent cache; >> parsing. >> 230208 13:32:32 22546 scitokens_Access: New valid token >> mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, >> issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska >> 230208 13:32:32 22546 scitokens_Access: Trying token-based access control >> 230208 13:32:32 22546 scitokens_Access: Cached token >> mapped_username=xrootd, subject=2deba9d1-9558-4963-95bc-75e993e3c82e, >> issuer=https://iam-escape.cloud.cnaf.infn.it/, groups=/escape,/escape/ska >> 230208 13:32:32 22546 ofs_open: unknown.1:25@localhost Unable to create >> /data/testfile-token_WITHauthz.repo; permission denied >> 230208 13:32:32 22546 unknown.1:25@localhost ofs_close: use=0 fn=dummy >> 230208 13:32:32 22546 XrootdXeq: unknown.1:25@localhost disc 0:00:00 >> (send failure) >> 230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: Poller 0 removing >> FD 25 >> 230208 13:32:32 22546 unknown.1:25@localhost Xrd_Poll: FD 25 detached >> from poller 0; num=0 >> [centos@xrootd ~]$ >> >> >> On 8 Feb 2023, at 12:42, Fabio Andrijauskas <[log in to unmask]> >> wrote: >> >> Hi, >> >> Can you send how are you requesting/creating the files? >> >> Try to add this to get all the possible logs: >> >> pfc.trace all >> ofs.trace all >> xrd.trace all -sched >> pss.setopt DebugLevel 5 >> scitokens.trace all >> >> Did you try to create the the path on the export path? >> >> Can you send your sci token config file? >> >> >> >> On Wed, Feb 8, 2023 at 2:25 AM Dejan Vitlacil <[log in to unmask]> >> wrote: >> >>> Hi Matt, >>> >>> I run out of ideas, so all suggestions are appreciated. >>> I think unix bits are in place. >>> >>> Cheers, >>> Dejan >>> >>> …… >>> [centos@xrootd ~]$ ls -lh / |grep data >>> drwxr-xr-x. 3 xrootd xrootd 84 Feb 6 15:22 data >>> [centos@xrootd ~]$ ls -lh /data >>> total 12K >>> drwxr-xr-x. 2 xrootd xrootd 6 Feb 6 14:13 escape >>> -rw-r--r--. 1 xrootd xrootd 77 Jan 25 13:21 four.txt >>> -rw-r--r--. 1 xrootd xrootd 168 Feb 6 15:22 testfile-token.repo >>> -rw-r--r--. 1 xrootd xrootd 168 Feb 2 14:18 testfile.repo >>> [centos@xrootd ~]$ >>> >>> >>> > On 8 Feb 2023, at 11:05, Doidge, Matt <[log in to unmask]> >>> wrote: >>> > >>> > Hello, >>> > This is a bit of a low-level suggestion, but can the xrootd unix user >>> write to /data? I had similar looking issues with a test xrootd server and >>> a known working config where I had forgotten to chown the exported path. >>> > >>> > My apologies for the noise if you've already checked this. >>> > >>> > Cheers, >>> > Matt >>> > >>> > ________________________________________ >>> > From: [log in to unmask] <[log in to unmask]> on >>> behalf of Dejan Vitlacil <[log in to unmask]> >>> > Sent: 08 February 2023 09:24 >>> > To: Oliver Freyermuth >>> > Cc: [log in to unmask] >>> > Subject: [External] Re: XRootD and tokens >>> > >>> > This email originated outside the University. Check before clicking >>> links or attachments. >>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> >> -- >> >> *--Fábio Andrijauskas* >> >> >> -- > > *--Fábio Andrijauskas* > > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1