LISTSERV 16.5 - XROOTD-DEV Archives

Stephan Lammel noted intermittent TLS errors reported by XRootD clients, affecting ETF availability testing. He, Bockjoo and I have been puzzling over it, but have not found the underlying issue.

The problem also was discussed in 1736, but it doesn't seem to be solved.

Description

Intermittently, the client is disconnected by the redirector with a TLS error immediately after login. This causes a 120 second (by default) delay until the client retries.
xrdfs $HOST query config version will demonstrate the problem
Client logs show TLS error rc=-1 ec=5 (error_syscall) errno=104. and Socket error while handshaking: [ERROR] TLS error
Server logs show only disc
Having IPv6 enabled seems to mask the issue, as the client will immediately try the other protocol

Reproducibility

Reliably appears on an EL7 local redirector with TLS enabled. Redirector was running xrootd-5.5.3-1.2.osg36.el7.x86_64.
Happens roughly 0.5-3% of the time
Appears between systems on a local network, so not expecting packet loss or similar network issues
Tried client versions v5.4.2 and v5.5.3 and behavior was the same

Looking at a packet capture of an error, the client and server complete a TLS v1.2 handshake. Then the client sends an kXR_login. The server immediately closes the TCP connection with a RST/ACK. Compared to a successful login, the packet capture to that point looks nearly identical, with only the pid changing.

Enabling heap debugging with mimalloc did not detect any errors. I've not yet tried valgrind, because that would be challenging to run on our production redirector. I'm at a loss for how to debug this further, and would appreciate ideas.

Best regards,
John

Server logs (for the first attempt)

230308 14:31:00.412570 9861 Xrd_Inet: Accepted connection on port 1094 from [log in to unmask]
230308 14:31:00.419385 9861 Xrd_ProtLoad: matched port 1094 protocol xroot
230308 14:31:00.419410 9861 anon.0:306@t3 Xrd_Poll: FD 306 attached to poller 1; num=122
230308 14:31:00.419486 9861 anon.0:306@t3 TLS_Accept: Accepting a TLS connection...
230308 14:31:00.429852 9861 XrdLinkXeq: anon.0:306@t3 connection upgraded to TLSv1.2
230308 14:31:00.430337 9861 XrootdXeq: anon.0:306@t3 disc 0:00:00
230308 14:31:00.430363 9861 anon.0:306@t3 Xrd_Poll: Poller 1 removing FD 306
230308 14:31:00.430375 9861 anon.0:306@t3 Xrd_Poll: FD 306 detached from poller 1; num=121

Client logs

[2023-03-08 14:31:00.398502 -0600][Debug  ][Utility           ] Initializing xrootd client version: v5.4.2
[2023-03-08 14:31:00.399069 -0600][Debug  ][Utility           ] Unable to process user config file: [ERROR] OS Error: no such file or directory
[2023-03-08 14:31:00.399226 -0600][Info   ][Utility           ] Env: Importing from shell XRD_NETWORKSTACK=IPv4 as networkstack
[2023-03-08 14:31:00.399280 -0600][Debug  ][PlugInMgr         ] Initializing plug-in manager...
[2023-03-08 14:31:00.399296 -0600][Debug  ][PlugInMgr         ] No default plug-in, loading plug-in configs...
[2023-03-08 14:31:00.399307 -0600][Debug  ][PlugInMgr         ] Processing plug-in definitions in /etc/xrootd/client.plugins.d...
[2023-03-08 14:31:00.399417 -0600][Debug  ][PlugInMgr         ] Processing plug-in definitions in /home/swanson/jthiltge/.xrootd/client.plugins.d...
[2023-03-08 14:31:00.399445 -0600][Debug  ][PlugInMgr         ] Unable to process directory /home/swanson/jthiltge/.xrootd/client.plugins.d: [ERROR] OS Error: no such file or directory
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] URL: xrootd-local.unl.edu
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.401150 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] URL: root://xrootd-local.unl.edu:1094/
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.401198 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] URL: root://xrootd-local.unl.edu:1094/
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.401230 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] URL: root://xrootd-local.unl.edu:1094/
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.401260 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.401311 -0600][Debug  ][App               ] Executing: query config version 
[2023-03-08 14:31:00.401324 -0600][Dump   ][App               ] Param #00: 'query'
[2023-03-08 14:31:00.401335 -0600][Dump   ][App               ] Param #01: 'config'
[2023-03-08 14:31:00.401344 -0600][Dump   ][App               ] Param #02: 'version'
[2023-03-08 14:31:00.401388 -0600][Dump   ][FileSystem        ] [[log in to unmask]:1094] Sending kXR_query (code: kXR_Qconfig, arg length: 8)
[2023-03-08 14:31:00.401411 -0600][Debug  ][Utility           ] Env: trying to get a non-existent string entry: pollerpreference
[2023-03-08 14:31:00.401428 -0600][Debug  ][Poller            ] Available pollers: built-in
[2023-03-08 14:31:00.401447 -0600][Debug  ][Poller            ] Attempting to create a poller according to preference: built-in
[2023-03-08 14:31:00.401460 -0600][Debug  ][Poller            ] Creating poller: built-in
[2023-03-08 14:31:00.401476 -0600][Debug  ][Poller            ] Creating and starting the built-in poller...
[2023-03-08 14:31:00.410943 -0600][Debug  ][Poller            ] Using 1 poller threads
[2023-03-08 14:31:00.410983 -0600][Debug  ][TaskMgr           ] Starting the task manager...
[2023-03-08 14:31:00.411048 -0600][Debug  ][TaskMgr           ] Task manager started
[2023-03-08 14:31:00.411067 -0600][Debug  ][JobMgr            ] Starting the job manager...
[2023-03-08 14:31:00.411217 -0600][Debug  ][JobMgr            ] Job manager started, 3 workers
[2023-03-08 14:31:00.411240 -0600][Debug  ][TaskMgr           ] Registering task: "FileTimer task" to be run at: [2023-03-08 14:31:00 -0600]
[2023-03-08 14:31:00.411253 -0600][Dump   ][XRootD            ] [xrootd-local.unl.edu:1094] Sending message kXR_query (code: kXR_Qconfig, arg length: 8)
[2023-03-08 14:31:00.411293 -0600][Debug  ][ExDbgMsg          ] [xrootd-local.unl.edu:1094] MsgHandler created: 0x1f92480 (message: kXR_query (code: kXR_Qconfig, arg length: 8) ).
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] URL: xrootd-local.unl.edu:1094
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.411351 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] URL: root://xrootd-local.unl.edu:1094
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] Protocol:  root
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] User Name: 
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] Password:  
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] Host Name: xrootd-local.unl.edu
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] Port:      1094
[2023-03-08 14:31:00.411390 -0600][Dump   ][Utility           ] Path:      
[2023-03-08 14:31:00.411415 -0600][Debug  ][PostMaster        ] Creating new channel to: root://xrootd-local.unl.edu:1094
[2023-03-08 14:31:00.411449 -0600][Debug  ][PostMaster        ] [xrootd-local.unl.edu:1094] Stream parameters: Network Stack: IPv4, Connection Window: 120, ConnectionRetry: 5, Stream Error Window: 1800
[2023-03-08 14:31:00.411478 -0600][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" to be run at: [2023-03-08 14:31:15 -0600]
[2023-03-08 14:31:00.411498 -0600][Dump   ][PostMaster        ] [xrootd-local.unl.edu:1094] Sending message kXR_query (code: kXR_Qconfig, arg length: 8) (0x1f8d4c0) through substream 0 expecting answer at 0
[2023-03-08 14:31:00.411983 -0600][Debug  ][PostMaster        ] [xrootd-local.unl.edu:1094] Found 1 address(es): [::ffff:129.93.239.164]:1094
[2023-03-08 14:31:00.412050 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Attempting connection to [::ffff:129.93.239.164]:1094
[2023-03-08 14:31:00.412107 -0600][Debug  ][Poller            ] Adding socket 0x1f928b0 to the poller
[2023-03-08 14:31:00.418996 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Async connection call returned
[2023-03-08 14:31:00.419048 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending out the initial hand shake + kXR_protocol
[2023-03-08 14:31:00.419105 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x58000950), 44 bytes
[2023-03-08 14:31:00.419317 -0600][Dump   ][XRootDTransport   ] [msg: 0x58000ab0] Expecting 8 bytes of message body
[2023-03-08 14:31:00.419333 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:31:00.419357 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 16 bytes
[2023-03-08 14:31:00.419374 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Got the server hand shake response (type: manager [], protocol version 511)
[2023-03-08 14:31:00.419398 -0600][Dump   ][XRootDTransport   ] [msg: 0x58000ab0] Expecting 8 bytes of message body
[2023-03-08 14:31:00.419409 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:31:00.419422 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 16 bytes
[2023-03-08 14:31:00.419438 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] kXR_protocol successful (type: manager [], protocol version 511)
[2023-03-08 14:31:00.419931 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending out kXR_login request, username: jthiltge, cgi: ?xrd.cc=us&xrd.tz=-6&xrd.appname=xrdfs&xrd.info=&xrd.hostname=t3.unl.edu&xrd.rn=v5.4.2, dual-stack: true, private IPv4: false, private IPv6: false
[2023-03-08 14:31:00.419955 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:31:00.419967 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:31:00.419994 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:31:15 -0600]
[2023-03-08 14:31:00.425877 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:31:00.429813 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:31:00.429968 -0600][Info   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake done.
[2023-03-08 14:31:00.430031 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x58000be0), 110 bytes
[2023-03-08 14:31:00.430378 -0600][Error  ][TlsMsg            ] [] TLS error rc=-1 ec=5 (error_syscall) errno=104.
[2023-03-08 14:31:00.430403 -0600][Error  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Socket error while handshaking: [ERROR] TLS error
[2023-03-08 14:31:00.430417 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Closing the socket
[2023-03-08 14:31:00.430432 -0600][Debug  ][Poller            ] <[::ffff:129.93.239.166]:17946><--><[::ffff:129.93.239.164]:1094> Removing socket from the poller
[2023-03-08 14:31:00.430517 -0600][Error  ][PostMaster        ] [xrootd-local.unl.edu:1094] elapsed = 0, pConnectionWindow = 120 seconds.
[2023-03-08 14:31:00.430533 -0600][Info   ][PostMaster        ] [xrootd-local.unl.edu:1094] Attempting reconnection in 120 seconds.
[2023-03-08 14:31:00.430546 -0600][Debug  ][TaskMgr           ] Registering task: "StreamConnectorTask for xrootd-local.unl.edu:1094" to be run at: [2023-03-08 14:33:00 -0600]
[2023-03-08 14:31:15.420987 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:31:15.421038 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:31:30 -0600]
[2023-03-08 14:31:15.421051 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:31:15.421065 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:31:30 -0600]
[2023-03-08 14:31:30.421996 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:31:30.422036 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:31:45 -0600]
[2023-03-08 14:31:30.422048 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:31:30.422062 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:31:45 -0600]
[2023-03-08 14:31:45.424318 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:31:45.424377 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:32:00 -0600]
[2023-03-08 14:31:45.424390 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:31:45.424404 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:32:00 -0600]
[2023-03-08 14:32:00.425379 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:32:00.425419 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:32:15 -0600]
[2023-03-08 14:32:00.425431 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:32:00.425446 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:32:15 -0600]
[2023-03-08 14:32:15.426948 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:32:15.427016 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:32:30 -0600]
[2023-03-08 14:32:15.427035 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:32:15.427053 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:32:30 -0600]
[2023-03-08 14:32:30.429204 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:32:30.429272 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:32:45 -0600]
[2023-03-08 14:32:30.429290 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:32:30.429300 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:32:45 -0600]
[2023-03-08 14:32:45.431482 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:32:45.431561 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:33:00 -0600]
[2023-03-08 14:32:45.431582 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:32:45.431598 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:33:00 -0600]
[2023-03-08 14:33:00.433974 -0600][Dump   ][TaskMgr           ] Running task: "StreamConnectorTask for xrootd-local.unl.edu:1094"
[2023-03-08 14:33:00.434472 -0600][Debug  ][PostMaster        ] [xrootd-local.unl.edu:1094] Found 1 address(es): [::ffff:129.93.239.164]:1094
[2023-03-08 14:33:00.434525 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Attempting connection to [::ffff:129.93.239.164]:1094
[2023-03-08 14:33:00.434571 -0600][Debug  ][Poller            ] Adding socket 0x1f928b0 to the poller
[2023-03-08 14:33:00.434604 -0600][Debug  ][TaskMgr           ] Done with task: "StreamConnectorTask for xrootd-local.unl.edu:1094"
[2023-03-08 14:33:00.434614 -0600][Dump   ][TaskMgr           ] Running task: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:33:00.434627 -0600][Dump   ][TaskMgr           ] Will rerun task "TickGeneratorTask for: root://xrootd-local.unl.edu:1094" at [2023-03-08 14:33:15 -0600]
[2023-03-08 14:33:00.434635 -0600][Dump   ][TaskMgr           ] Running task: "FileTimer task"
[2023-03-08 14:33:00.434645 -0600][Dump   ][TaskMgr           ] Will rerun task "FileTimer task" at [2023-03-08 14:33:15 -0600]
[2023-03-08 14:33:00.435004 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Async connection call returned
[2023-03-08 14:33:00.435052 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending out the initial hand shake + kXR_protocol
[2023-03-08 14:33:00.435106 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x58042950), 44 bytes
[2023-03-08 14:33:00.435414 -0600][Dump   ][XRootDTransport   ] [msg: 0x58041170] Expecting 8 bytes of message body
[2023-03-08 14:33:00.435444 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:33:00.435468 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 16 bytes
[2023-03-08 14:33:00.435479 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Got the server hand shake response (type: manager [], protocol version 511)
[2023-03-08 14:33:00.435514 -0600][Dump   ][XRootDTransport   ] [msg: 0x5803dce0] Expecting 8 bytes of message body
[2023-03-08 14:33:00.435524 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:33:00.435535 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 16 bytes
[2023-03-08 14:33:00.435547 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] kXR_protocol successful (type: manager [], protocol version 511)
[2023-03-08 14:33:00.436589 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending out kXR_login request, username: jthiltge, cgi: ?xrd.cc=us&xrd.tz=-6&xrd.appname=xrdfs&xrd.info=&xrd.hostname=t3.unl.edu&xrd.rn=v5.4.2, dual-stack: true, private IPv4: false, private IPv6: false
[2023-03-08 14:33:00.436609 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:33:00.440214 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:33:00.442756 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake exchange.
[2023-03-08 14:33:00.442864 -0600][Info   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] TLS hand-shake done.
[2023-03-08 14:33:00.442907 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x580008c0), 110 bytes
[2023-03-08 14:33:00.443289 -0600][Dump   ][XRootDTransport   ] [msg: 0x58043690] Expecting 75 bytes of message body
[2023-03-08 14:33:00.443310 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:33:00.443379 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 83 bytes
[2023-03-08 14:33:00.443399 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Logged in, session: da28000048260000a5000000562d0000
[2023-03-08 14:33:00.443408 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Authentication is required: &P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096:
[2023-03-08 14:33:00.443418 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending authentication data
[2023-03-08 14:33:00.444765 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Trying to authenticate using gsi
[2023-03-08 14:33:01.330423 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x58061520), 136 bytes
[2023-03-08 14:33:01.338711 -0600][Dump   ][XRootDTransport   ] [msg: 0x5803d430] Expecting 3252 bytes of message body
[2023-03-08 14:33:01.338741 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:33:01.338807 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 3260 bytes
[2023-03-08 14:33:01.338829 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Sending more authentication data for gsi
[2023-03-08 14:33:01.349005 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message:  (0x58064440), 9932 bytes
[2023-03-08 14:33:01.398007 -0600][Dump   ][XRootDTransport   ] [msg: 0x58043690] Expecting 0 bytes of message body
[2023-03-08 14:33:01.398037 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header, size: 8
[2023-03-08 14:33:01.398055 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received a message of 8 bytes
[2023-03-08 14:33:01.398129 -0600][Debug  ][XRootDTransport   ] [xrootd-local.unl.edu:1094.0] Authenticated with gsi.
[2023-03-08 14:33:01.398149 -0600][Debug  ][PostMaster        ] [xrootd-local.unl.edu:1094] Stream 0 connected.
[2023-03-08 14:33:01.398163 -0600][Debug  ][Utility           ] Monitor library name not set. No monitoring
[2023-03-08 14:33:01.398193 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Wrote a message: kXR_query (code: kXR_Qconfig, arg length: 8) (0x1f8d4c0), 32 bytes
[2023-03-08 14:33:01.398219 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Successfully sent message: kXR_query (code: kXR_Qconfig, arg length: 8) (0x1f8d4c0).
[2023-03-08 14:33:01.398231 -0600][Dump   ][XRootD            ] [xrootd-local.unl.edu:1094] Message kXR_query (code: kXR_Qconfig, arg length: 8) has been successfully sent.
[2023-03-08 14:33:01.398240 -0600][Debug  ][ExDbgMsg          ] [xrootd-local.unl.edu:1094] Moving MsgHandler: 0x1f92480 (message: kXR_query (code: kXR_Qconfig, arg length: 8) ) from out-queu to in-queue.
[2023-03-08 14:33:01.398252 -0600][Dump   ][PostMaster        ] [xrootd-local.unl.edu:1094.0] All messages consumed, disable uplink
[2023-03-08 14:33:01.398620 -0600][Dump   ][XRootDTransport   ] [msg: 0x580008d0] Expecting 7 bytes of message body
[2023-03-08 14:33:01.398651 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message header for 0x580008d0 size: 8
[2023-03-08 14:33:01.398708 -0600][Dump   ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Received message 0x580008d0 of 15 bytes
[2023-03-08 14:33:01.398731 -0600][Dump   ][PostMaster        ] [xrootd-local.unl.edu:1094] Handling received message: 0x580008d0.
[2023-03-08 14:33:01.398832 -0600][Dump   ][XRootD            ] [xrootd-local.unl.edu:1094] Got a kXR_ok response to request kXR_query (code: kXR_Qconfig, arg length: 8)
[2023-03-08 14:33:01.398867 -0600][Debug  ][ExDbgMsg          ] [xrootd-local.unl.edu:1094] Calling MsgHandler: 0x1f92480 (message: kXR_query (code: kXR_Qconfig, arg length: 8) ) with status: [SUCCESS] .
[2023-03-08 14:33:01.398891 -0600][Dump   ][XRootD            ] [xrootd-local.unl.edu:1094] Parsing the response to kXR_query (code: kXR_Qconfig, arg length: 8) as BinaryData
[2023-03-08 14:33:01.398925 -0600][Dump   ][FileSystem        ] [[log in to unmask]:1094] Assigning xrootd-local.unl.edu:1094 as last URL
[2023-03-08 14:33:01.398948 -0600][Debug  ][ExDbgMsg          ] [xrootd-local.unl.edu:1094] Destroying MsgHandler: 0x1f92480.
v5.5.3

[2023-03-08 14:33:01.400842 -0600][Debug  ][JobMgr            ] Stopping the job manager...
[2023-03-08 14:33:01.400867 -0600][Dump   ][JobMgr            ] Stopping worker #0...
[2023-03-08 14:33:01.401091 -0600][Dump   ][JobMgr            ] Worker #0 stopped
[2023-03-08 14:33:01.401123 -0600][Dump   ][JobMgr            ] Stopping worker #1...
[2023-03-08 14:33:01.401330 -0600][Dump   ][JobMgr            ] Worker #1 stopped
[2023-03-08 14:33:01.401362 -0600][Dump   ][JobMgr            ] Stopping worker #2...
[2023-03-08 14:33:01.401464 -0600][Dump   ][JobMgr            ] Worker #2 stopped
[2023-03-08 14:33:01.401496 -0600][Debug  ][JobMgr            ] Job manager stopped
[2023-03-08 14:33:01.401519 -0600][Debug  ][TaskMgr           ] Stopping the task manager...
[2023-03-08 14:33:01.401784 -0600][Debug  ][TaskMgr           ] Task manager stopped
[2023-03-08 14:33:01.401817 -0600][Debug  ][Poller            ] Stopping the poller...
[2023-03-08 14:33:01.401963 -0600][Debug  ][TaskMgr           ] Requesting unregistration of: "TickGeneratorTask for: root://xrootd-local.unl.edu:1094"
[2023-03-08 14:33:01.401994 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Closing the socket
[2023-03-08 14:33:01.402021 -0600][Debug  ][Poller            ] <[::ffff:129.93.239.166]:7838><--><[::ffff:129.93.239.164]:1094> Removing socket from the poller
[2023-03-08 14:33:01.402159 -0600][Debug  ][PostMaster        ] [xrootd-local.unl.edu:1094] Destroying stream
[2023-03-08 14:33:01.402183 -0600][Debug  ][AsyncSock         ] [xrootd-local.unl.edu:1094.0] Closing the socket

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/xrootd/xrootd/issues/1952", "url": "https://github.com/xrootd/xrootd/issues/1952", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1