After upgrading from v5.5.0 to v5.5.1, we're seeing our EL7 local redirector stop accepting GFAL davs writes after 8 hours. I suspect the issue is associated with the changes in #1796 (or other recent changes to certificate refresh). After the redirector xrootd daemon has run 8 hours, the TLS v1.2 server hello includes "SessionTicket TLS", and it provides session tickets to clients. But if a client attempts to (re)start a TLS session using the ticket, the redirector immediately closes the connection. Before the refresh interval, the redirector did not report TLS session ticket support. A restart of the xrootd service returns it to normal, until the next 8h interval has passed.
Local redirector config comments:
- `xrootd.tls capable all -data`
- `xrd.tlsca noverify`
- `http.tlsreuse` is not set, using the default of `off`
- RPMs
- `xrootd-5.5.1-1.4.osg36.el7.x86_64`
- `openssl-1.0.2k-25.el7_9.x86_64`
- No cluster filesystem mounted. `cms.dfs lookup distrib redirect immed`
- Macaroons enabled
davs write to show issue on an affected redirector:
```
gfal-copy source davs://local-redirector.example.edu:1094/target
```
<details>
<summary>gfal-copy log excerpt showing failure</summary>
```
2022-12-18T11:15:56.092990 DEBUG Davix: NEON start internal request
2022-12-18T11:15:56.093083 DEBUG Davix: cached ne_session found ! taken from cache
2022-12-18T11:15:56.093165 DEBUG Davix: configure session...
2022-12-18T11:15:56.103282 DEBUG Davix: define connection timeout to 30
2022-12-18T11:15:56.103361 DEBUG Davix: define operation timeout to 3600
2022-12-18T11:15:56.103415 DEBUG Davix: add CA PATH /etc/grid-security/certificates/
2022-12-18T11:15:56.103464 DEBUG Davix: disable login/password authentication
2022-12-18T11:15:56.103616 DEBUG Davix: enable client cert authentication by callback
2022-12-18T11:15:56.103696 DEBUG Davix: Disable Session recycling
2022-12-18T11:15:56.103750 DEBUG Davix: Running pre_send hooks
2022-12-18T11:15:56.103836 DEBUG Davix: > PUT /store/user/jthiltge/test.1671383724.975305632 HTTP/1.1
2022-12-18T11:15:56.103904 > User-Agent: gfal2-util/1.8.0 gfal2/2.21.1 neon/0.0.29
2022-12-18T11:15:56.103972 > TE: trailers
2022-12-18T11:15:56.104022 > Host: xrootd-local.unl.edu:1094
2022-12-18T11:15:56.104052 > Authorization: Bearer [...token...]
2022-12-18T11:15:56.104082 > Content-Length: 1168
2022-12-18T11:15:56.104111 > Expect: 100-continue
2022-12-18T11:15:56.104140 >
2022-12-18T11:15:56.104168
2022-12-18T11:15:56.104197 DEBUG Davix: Sending request-line and headers:
2022-12-18T11:15:56.104224 DEBUG Davix: Request sent; retry is 1.
2022-12-18T11:15:56.104263 DEBUG Davix: Aborted request (-3): Could not read status line
2022-12-18T11:15:56.104294 DEBUG Davix: sess: Closing connection.
2022-12-18T11:15:56.104328 DEBUG Davix: sess: Connection closed.
2022-12-18T11:15:56.104449 DEBUG Davix: Persistent connection timed out, retrying.
2022-12-18T11:15:56.104536 DEBUG Davix: Sending request-line and headers:
2022-12-18T11:15:56.104954 DEBUG Davix: Doing SSL negotiation.
2022-12-18T11:16:27.002101 DEBUG Davix: sess: Closing connection.
2022-12-18T11:16:27.002202 DEBUG Davix: sess: Connection closed.
2022-12-18T11:16:27.002323 DEBUG Davix: Disable Session recycling
2022-12-18T11:16:27.002473 DEBUG Davix: sess: Not closing closed connection.
2022-12-18T11:16:27.002531 DEBUG Davix: Disable Session recycling
2022-12-18T11:16:27.002622 DEBUG Davix: Connection problem: eradicate session
2022-12-18T11:16:27.002680 DEBUG Davix: Disable Session recycling
2022-12-18T11:16:27.002839 DEBUG Davix: <- negotiateRequest
2022-12-18T11:16:27.002896 DEBUG Davix: <- executeRequest
2022-12-18T11:16:27.003025 DEBUG Davix: write result size 1168
2022-12-18T11:16:27.003282 DEBUG Davix: Destroy HttpRequest
2022-12-18T11:16:27.003344 DEBUG Davix: Running destroy hooks.
2022-12-18T11:16:27.003409 DEBUG Davix: Request ends.
2022-12-18T11:16:27.003552 DEBUG Davix: sess: Destroying session.
2022-12-18T11:16:27.005239 DEBUG <- gfal_plugin_closeG
2022-12-18T11:16:27.005615 WARNING Copy failed with mode streamed: [gfal_http_streamed_copy] (Neon): SSL handshake failed: Connection timed out during SSL handshake (destination)
```
</details>
The gfal/davix conversation to PUT a file involves a number of requests to the redirector and servers. After the redirector replies to a HEAD request, it closes the TCP connection (despite responding as `Connection: Keep-Alive` which is another minor issue.) When davix tries to reuse the connection to the redirector, it finds the connection closed, and reconnects. This is all fine so far. But if the client attempts to reestablish the connection using a TLS session ticket from the redirector, the connection is immediately dropped, and the file upload fails.
I'm guessing that when the TLS context is refreshed, the session ticket parameters are not set on the new context. Oddly, I'm only seeing this behavior on our EL7 redirector, but not the EL8 data servers.
The issue can also be shown with `s_client`:
```
openssl s_client -tls1_2 -state -reconnect xrootd-local.unl.edu:1094
```
<details>
<summary>s_client output for affected redirector</summary>
```
$ openssl s_client -tls1_2 -state -reconnect xrootd-local.unl.edu:1094
CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = US, O = Internet2, OU = InCommon, CN = InCommon IGTF Server CA
verify return:1
depth=0 DC = org, DC = incommon, C = US, ST = Nebraska, O = University of Nebraska-Lincoln, CN = xrootd-local.unl.edu
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read server session ticket
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
---
Certificate chain
0 s:DC = org, DC = incommon, C = US, ST = Nebraska, O = University of Nebraska-Lincoln, CN = xrootd-local.unl.edu
i:C = US, O = Internet2, OU = InCommon, CN = InCommon IGTF Server CA
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 12 00:00:00 2022 GMT; NotAfter: Aug 11 23:59:59 2023 GMT
1 s:C = US, O = Internet2, OU = InCommon, CN = InCommon IGTF Server CA
i:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA512
v:NotBefore: Jan 1 00:00:00 2014 GMT; NotAfter: Dec 31 23:59:59 2023 GMT
2 s:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
i:C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384
v:NotBefore: Jan 19 00:00:00 2010 GMT; NotAfter: Jan 18 23:59:59 2038 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGlzCCBX+gAwIBAgIRALwaOcSNiaoxvJrp6bn33ZwwDQYJKoZIhvcNAQELBQAw
VjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUludGVybmV0MjERMA8GA1UECxMISW5D
b21tb24xIDAeBgNVBAMTF0luQ29tbW9uIElHVEYgU2VydmVyIENBMB4XDTIyMDcx
MjAwMDAwMFoXDTIzMDgxMTIzNTk1OVowgZcxEzARBgoJkiaJk/IsZAEZFgNvcmcx
GDAWBgoJkiaJk/IsZAEZFghpbmNvbW1vbjELMAkGA1UEBhMCVVMxETAPBgNVBAgT
CE5lYnJhc2thMScwJQYDVQQKEx5Vbml2ZXJzaXR5IG9mIE5lYnJhc2thLUxpbmNv
bG4xHTAbBgNVBAMTFHhyb290ZC1sb2NhbC51bmwuZWR1MIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAzmyvHnXcNyPKsQuV45WBT1jIna3xrjURsbJwMX5B
0SRDC+a+20u/Im8U0JeYqXKUWEMyQLHJItSjNqyYl6SeeuXn+wjnvr17Yn9jMLET
2pi5hV0XUbLIvDn6hHC4Jm8/sjkwinlLNnKnRzmiwbwld5OoVxvwWCFDCZbGUnn2
udb1YUwhYL9Ths5ZPoq1USAPiUAPSivPl7MUPqs7SxTMycRgXKDroKjLhdBIwZDN
DTyUOD5jyR3DKe9+TioSuLbq3Ka+CrrgjEl/T7Q8s7B+q78gwo9y+ARUykOGs20A
rW4AUwYMvzXeB2YvyaDnGs36hDQHArlcEkMw+rn13HhYNwIDAQABo4IDHDCCAxgw
HwYDVR0jBBgwFoAUYeZfySe6QPjsKAbziTJaVl/ijtEwHQYDVR0OBBYEFG4v2p1D
OjOZDrUa7ldHtnwICkEYMA4GA1UdDwEB/wQEAwIEsDAMBgNVHRMBAf8EAjAAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAyBgNVHSAEKzApMA8GDSsGAQQB
riMBBAMEAQEwDAYKKoZIhvdMBQICATAIBgZngQwBAgIwRgYDVR0fBD8wPTA7oDmg
N4Y1aHR0cDovL2NybC5pbmNvbW1vbi1pZ3RmLm9yZy9JbkNvbW1vbklHVEZTZXJ2
ZXJDQS5jcmwwfAYIKwYBBQUHAQEEcDBuMEEGCCsGAQUFBzAChjVodHRwOi8vY3J0
LmluY29tbW9uLWlndGYub3JnL0luQ29tbW9uSUdURlNlcnZlckNBLmNydDApBggr
BgEFBQcwAYYdaHR0cDovL29jc3AuaW5jb21tb24taWd0Zi5vcmcwggF8BgorBgEE
AdZ5AgQCBIIBbASCAWgBZgB1AK33vvp8/xDIi509nB4+GGq0Zyldz7EMJMqFhjTr
3IKKAAABgfRdNrMAAAQDAEYwRAIgWcDBSylMEDxJ/kMJZZ1gqcl2+gLGpOxMw/0K
fLPh9y0CIDFuovTl9+gGA/epSzZ2Vi5ZYJq9Tb+Lzg9n4PqqVUuzAHUAejKMVNi3
LbYg6jjgUh7phBZwMhOFTTvSK8E6V6NS61IAAAGB9F02hwAABAMARjBEAiA0JGoa
7yykxGl/eu5X7JdpHaGPmU5TbrNoT86GmUdWqQIgYZ1qbIFuJ86qPUR1q+eNzirc
JyCn9M7VchNmVoB+ryQAdgDoPtDaPvUGNTLnVyi8iWvJA9PL0RFr7Otp4Xd9bQa9
bgAAAYH0XTZPAAAEAwBHMEUCIQC0PB4SNcv4Ujo7Cu508c4wZ/h0Y/cbAXEF8uGU
1hr1dgIgBsclAyiQddcobwWY6E+sqaS3XiLmD9lq1gWKIge3m44wHwYDVR0RBBgw
FoIUeHJvb3RkLWxvY2FsLnVubC5lZHUwDQYJKoZIhvcNAQELBQADggEBAA/cjqvW
UDC6my+IIQX57gg6oGqOO4xktsZlW3lnDOb29T8zX3+n6bmzYP2a2aITluBKiB7Z
1QwgnmmDGVxzPr8YNcsBYz1r/HYfUb/VwNNdsDnSUHjK4m5vKq2wJJuOylTYPaAW
qgUisj2v7M/km85Pg1nVJPceVoyfVU6MDifwNtyhUmVqRRUeaPqGHu1fpdJeC92k
5NZOZHjjMWWEkWyHWyOZeHgg1pWxBDKmPCiW8snCa6W1/b1ec9jAZqIolKetjjnW
vj3bKv86cxutrbceB7lS3kZ6jOLJw3AU+ciPFu09q5ZM54lz74xQKnnYbtnDvbXU
TPKcV0qTefw7P4s=
-----END CERTIFICATE-----
subject=DC = org, DC = incommon, C = US, ST = Nebraska, O = University of Nebraska-Lincoln, CN = xrootd-local.unl.edu
issuer=C = US, O = Internet2, OU = InCommon, CN = InCommon IGTF Server CA
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA384:ECDSA+SHA384:RSA+SHA256:ECDSA+SHA256:RSA+SHA224:ECDSA+SHA224
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 5422 bytes and written 355 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1B602353DAAF811A602A2B3F30542C57D0801ED0335489C65C37416A4AB1DFD2
Session-ID-ctx:
Master-Key: AB44B96FFEC1AB2058411B3F69E8540D2048179931B5B83F26CCBA259A0F6214C0D4F61650B14B999122456FBAA07BC7
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 17 b7 54 40 db 73 b1 b9-b7 a6 e6 7b 93 8d 1f e3 ..T@.s.....{....
0010 - ee 35 08 8b a8 01 d4 54-93 6d f1 0b 40 b1 3c d9 [log in to unmask]<.
0020 - fb 88 9a 1d 98 64 70 9b-17 1f 43 b2 7e 62 1b e7 .....dp...C.~b..
0030 - 92 21 2e 37 79 64 fb 81-56 19 3c b9 bd 95 00 c6 .!.7yd..V.<.....
0040 - df 1a dc d3 7d 7a 77 70-08 77 80 2b 69 6d b6 80 ....}zwp.w.+im..
0050 - ef 4c 43 3d a0 66 ae 97-b3 2a 01 c9 a3 e3 56 e4 .LC=.f...*....V.
0060 - b4 ca 95 65 34 c1 ca 4e-7c 4e 96 1d d9 a8 30 57 ...e4..N|N....0W
0070 - b1 8e f9 53 10 45 fd 06-34 7e bc 2e 02 8a 15 71 ...S.E..4~.....q
0080 - 24 90 91 63 03 92 08 35-eb 75 80 2e f1 d0 e2 88 $..c...5.u......
0090 - 52 ba 6f 05 3f c6 b0 ed-8d 37 99 e8 a0 97 85 4d R.o.?....7.....M
00a0 - 9d 6f 3e 16 87 e4 c1 37-8a b0 75 54 f4 c8 d4 9f .o>....7..uT....
Start Time: 1671415353
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
drop connection and then reconnect
SSL3 alert write:warning:close notify
CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL3 alert write:fatal:decode error
SSL_connect:error in error
800BCD4EBF7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:308:
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1B602353DAAF811A602A2B3F30542C57D0801ED0335489C65C37416A4AB1DFD2
Session-ID-ctx:
Master-Key: AB44B96FFEC1AB2058411B3F69E8540D2048179931B5B83F26CCBA259A0F6214C0D4F61650B14B999122456FBAA07BC7
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 17 b7 54 40 db 73 b1 b9-b7 a6 e6 7b 93 8d 1f e3 ..T@.s.....{....
0010 - ee 35 08 8b a8 01 d4 54-93 6d f1 0b 40 b1 3c d9 [log in to unmask]<.
0020 - fb 88 9a 1d 98 64 70 9b-17 1f 43 b2 7e 62 1b e7 .....dp...C.~b..
0030 - 92 21 2e 37 79 64 fb 81-56 19 3c b9 bd 95 00 c6 .!.7yd..V.<.....
0040 - df 1a dc d3 7d 7a 77 70-08 77 80 2b 69 6d b6 80 ....}zwp.w.+im..
0050 - ef 4c 43 3d a0 66 ae 97-b3 2a 01 c9 a3 e3 56 e4 .LC=.f...*....V.
0060 - b4 ca 95 65 34 c1 ca 4e-7c 4e 96 1d d9 a8 30 57 ...e4..N|N....0W
0070 - b1 8e f9 53 10 45 fd 06-34 7e bc 2e 02 8a 15 71 ...S.E..4~.....q
0080 - 24 90 91 63 03 92 08 35-eb 75 80 2e f1 d0 e2 88 $..c...5.u......
0090 - 52 ba 6f 05 3f c6 b0 ed-8d 37 99 e8 a0 97 85 4d R.o.?....7.....M
00a0 - 9d 6f 3e 16 87 e4 c1 37-8a b0 75 54 f4 c8 d4 9f .o>....7..uT....
Start Time: 1671415353
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
```
</details>
--
Reply to this email directly or view it on GitHub:
https://github.com/xrootd/xrootd/issues/1874
You are receiving this because you are subscribed to this thread.
Message ID: <[log in to unmask]>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
