[log in to unmask]">On Sun, 17 Mar 2024, Petr Vokac wrote:
On 3/15/24 16:42, Marcus Ebert wrote:Right. Redirector and servers are in different subnets and there have been issues with the routing using IPv6. For now we disabled all IPv6 and removed the DNS entry for it, focusing only on IPv4 to see if that works. Later on we can enable IPv6 again once all routing issues are fixed.
One correction:
On Fri, 15 Mar 2024, Marcus Ebert wrote:
Setup isservers are xrd{4..6}.belle.uvic.ca
- redirector rdc-redirector.belle.uvic.ca
- 3 servers xrd{1..3}.belle.uvic.ca
Simple macaroon token request takes 30s?! You should probably start with this issue ... actually this is pretty simple, your IPv6 connectivity is broken and this could bring with a lot of fun especially with old HTTP libraries like libneon that are still in by KEK FTS server (actually for better IPv6 support it would be necessary to enable `curl` for FTS/gfal and run FTS on EL8 or EL9 and that OS version is not yet officially supported).
You should really fix HTTP-TPC pull mode, which should be preferred (btw: FTS developers now considering changes in FTS that will prevent automatic fallback push <-> pull in case one method doesn't work, because this functionality just hides serious site issues and makes HTTP-TPC transfers debugging unnecessary complex)
Indeed, removing IPv6 and the DNS entry for it makes pull work now. At least in most cases. Only issue that remains is with a KEK server while all other sites can transfer from that KEK server just fine.
The pull still doesn't work with that server and for push FTS gets a " SocketException while pushing.... Connection reset by peer (Write failed)". It doesn't happen for all transfers but for most.
An example fts log for this kind of error can be found at
https://particle.phys.uvic.ca/~mebert/xrootd/fts-kek.txt
and the parts of the local log are in
https://particle.phys.uvic.ca/~mebert/xrootd/redirector.log
https://particle.phys.uvic.ca/~mebert/xrootd/xrd6.log
https://particle.phys.uvic.ca/~mebert/xrootd/xrd5.log
(in case things are missing in the logs, please let me know how I can get the output corresponding to a single request)
KEK is using StoRM, could you try your tests with other StoRM source?
You could try to download file with curl or even initiate
HTTP-TPC manually, e.g.
export SRC=https://belle2-webdav-raw-data.cc.kek.jp:8443/belle/TMP/belle/Raw/e0026/physics/r00194/sub00/physics.0026.00194.HLT2.f00002.root
export DST=https://rdc-redirector.belle.uvic.ca:1094/TMP/belle/some_test_file
export TSRC=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT30M"}' "$SRC" | jq -r '.macaroon')
export TDST=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST"], "validity": "PT30M"}' "$DST" | jq -r '.macaroon')
# StoRM is using JWT tokens, so you can inspect token from KEK site using e.g. https://jwt.io
# it is possible to inspect also macaroon tokens returned by XRootD with e.g.
# python -c "import macaroons; print(macaroons.deserialize('$TDST'[12:]).inspect())"
# Download
curl -v --capath /etc/grid-security/certificates -L -X GET -H 'Secure-Redirection: 1' -H "Authorization: Bearer $TSRC" --output /tmp/x "$SRC"
# Upload
curl -v --capath /etc/grid-security/certificates -L -X PUT -H 'Secure-Redirection: 1' -H "Authorization: Bearer $TDST" --upload-file /etc/services "$DST"
# TPC pull
curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
# TPC push
curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' -H "Authorization: Bearer $TSRC" -H "TransferHeaderAuthorization: Bearer $TDST" -H "Destination: $DST" "$SRC"
Petr