On 3/19/24 17:40, Marcus Ebert wrote:
> On Sun, 17 Mar 2024, Petr Vokac wrote:
>
>> On 3/15/24 16:42, Marcus Ebert wrote:
>>> One correction:
>>>
>>> On Fri, 15 Mar 2024, Marcus Ebert wrote:
>>>
>>>> Setup is
>>>> - redirector rdc-redirector.belle.uvic.ca
>>>> - 3 servers xrd{1..3}.belle.uvic.ca
>>>>
>>> servers are xrd{4..6}.belle.uvic.ca
>>
>> Simple macaroon token request takes 30s?! You should probably start
>> with this issue ... actually this is pretty simple, your IPv6
>> connectivity is broken and this could bring with a lot of fun
>> especially with old HTTP libraries like libneon that are still in by
>> KEK FTS server (actually for better IPv6 support it would be
>> necessary to enable `curl` for FTS/gfal and run FTS on EL8 or EL9 and
>> that OS version is not yet officially supported).
>>
>> You should really fix HTTP-TPC pull mode, which should be preferred
>> (btw: FTS developers now considering changes in FTS that will prevent
>> automatic fallback push <-> pull in case one method doesn't work,
>> because this functionality just hides serious site issues and makes
>> HTTP-TPC transfers debugging unnecessary complex)
>>
> Right. Redirector and servers are in different subnets and there have
> been issues with the routing using IPv6. For now we disabled all IPv6
> and removed the DNS entry for it, focusing only on IPv4 to see if that
> works. Later on we can enable IPv6 again once all routing issues are
> fixed.
>
>
> Indeed, removing IPv6 and the DNS entry for it makes pull work now. At
> least in most cases. Only issue that remains is with a KEK server
> while all other sites can transfer from that KEK server just fine.
>
> The pull still doesn't work with that server and for push FTS gets a "
> SocketException while pushing.... Connection reset by peer (Write
> failed)". It doesn't happen for all transfers but for most.
>
> An example fts log for this kind of error can be found at
> https://particle.phys.uvic.ca/~mebert/xrootd/fts-kek.txt
> and the parts of the local log are in
> https://particle.phys.uvic.ca/~mebert/xrootd/redirector.log
> https://particle.phys.uvic.ca/~mebert/xrootd/xrd6.log
> https://particle.phys.uvic.ca/~mebert/xrootd/xrd5.log
> (in case things are missing in the logs, please let me know how I can
> get the output corresponding to a single request)
KEK is using StoRM, could you try your tests with other StoRM source?
You could try to download file with curl or even initiate HTTP-TPC
manually, e.g.
export SRC=https://belle2-webdav-raw-data.cc.kek.jp:8443/belle/TMP/belle/Raw/e0026/physics/r00194/sub00/physics.0026.00194.HLT2.f00002.root
export DST=https://rdc-redirector.belle.uvic.ca:1094/TMP/belle/some_test_file
export TSRC=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT30M"}' "$SRC" | jq -r '.macaroon')
export TDST=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST"], "validity": "PT30M"}' "$DST" | jq -r '.macaroon')
# StoRM is using JWT tokens, so you can inspect token from KEK site using e.g.https://jwt.io
# it is possible to inspect also macaroon tokens returned by XRootD with e.g.
# python -c "import macaroons; print(macaroons.deserialize('$TDST'[12:]).inspect())"
# Download
curl -v --capath /etc/grid-security/certificates -L -X GET -H 'Secure-Redirection: 1' -H "Authorization: Bearer $TSRC" --output /tmp/x "$SRC"
# Upload
curl -v --capath /etc/grid-security/certificates -L -X PUT -H 'Secure-Redirection: 1' -H "Authorization: Bearer $TDST" --upload-file /etc/services "$DST"
# TPC pull
curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
# TPC push
curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' -H "Authorization: Bearer $TSRC" -H "TransferHeaderAuthorization: Bearer $TDST" -H "Destination: $DST" "$SRC"
Petr
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
