Sorry for the similarly late response.


I believe this is a 4.8.4 client with a 4.9.1 server. I had disregarded that line as we see that at the start of every transfer, even successful ones, so I assumed it was a warning that could be ignored. I may be completely wrong on this however!


I’m no longer particularly convinced these failures are indicative of a problem with the XRootD server. As Andy pointed out to me, a broken pipe send failure in the logs can be caused by something as simple as the client being killed during the transfer.


Unfortunately the failing jobs are user analysis jobs, so the failures are presented as somewhat cryptic (to me anyway!) experiment framework errors, and the entirely transient nature of these failures means that we haven’t been able to reproduce the failures. We’re working with the VO to try and isolate the issue better.


Hopefully I’ll be able to update you all with the resolution, or more useful error messages as we get a better handle on the issue.





From: Michal Kamil Simon <[log in to unmask]>
Sent: 05 August 2019 11:39
To: Byrne, Thomas (STFC,RAL,SC) <[log in to unmask]>; [log in to unmask]
Subject: RE: Help understanding xrootd broken pipe failures


Hi Tom,


Sorry for the late response, from the logs it looks like a post 4.9.x client tries to delegate

a credential to a pre 4.9.x server. Could you check what are the versions of the server and






From: [log in to unmask] [[log in to unmask]] on behalf of Thomas Byrne - UKRI STFC [[log in to unmask]]
Sent: 25 July 2019 12:02
[log in to unmask]
Subject: Help understanding xrootd broken pipe failures

Hi all,


Sorry for the vague nature of this email, I’m having significant issues understanding and tracking down the root cause of this problem, so any help/advice would be appreciated!


We are experiencing a large number of ‘broken pipe’ events causing job failures, mainly on files opened for streaming. For example:


190724 14:04:36 25445 secgsi_ServerDoCert: no signed DH parameters from client:tlhcb005.201:[log in to unmask] : will not delegate x509 proxy to it

190724 14:04:36 25445 XrootdXeq: tlhcb005.201:[log in to unmask] pvt IPv4 login as lhcbuser

190724 14:05:30 29851 acc_Audit: tlhcb005.201:[log in to unmask] grant gsi [log in to unmask] read lhcb:prod/lhcb/MC/2016/ALLSTREAMS.DST/00086001/0000/00086001_00000076_7.AllStreams.dst

190724 14:07:12 29851 XrdLink: Unable to send to tlhcb005.201:[log in to unmask]; broken pipe

190724 14:07:12 29851 XrootdXeq: tlhcb005.201:[log in to unmask] disc 0:02:37 (send failure)


We are seeing this on XRootD server containers on our worker nodes, with jobs running on the same worker node. The file may coming from the local disk cache, or from another local XRootD server container which is talking to a Ceph cluster via libxrdceph.


Has anyone seen this before? Given the specific error message I am assuming that there is some communication issue, but I was wondering if anyone could shed some light on what exactly is breaking here, or how I can get more detail about the issue (debugging settings etc.).





Tom Byrne

Storage System Administrator

Scientific Computing Department

Science and Technology Facilities Council

Rutherford Appleton Laboratory



Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link: