Currently, there is no way to get cache statistics in the 4.x series.
Limited statistics can be generated in the 5.x series. However, you will
need to wait for a patch release to run the 5.x series with Xcache.
On Fri, 24 Jul 2020, Tom Byrne wrote:
> I've extracted the respective server log file for the Xcache and Ceph gateway for the job log snippet in Raja's first post. I've had to clean this up fairly heavily as it all ends up in /var/log/messages, but I think I've got the relevant bits. The timeout in the original post happened at 7:51:18 in this log.
>
> I don't think there is anything particularly suspicious, although I may be missing something.
>
> Next I'm going to try harmonising the block sizes between the Xcache and the gateway (to 64MB), and I'm going to turn on the relevant xrootd tracing setting so we can see what vector reads the servers are trying to carry out when these errors occur. I'll also look into whether Xcache can report any cache efficiency stats, I know the memory proxy has an option to log various cache stats, but I looked for anything similar for Xcache.
>
>> Jul 22 07:44:43 lcg2160 docker: 200722 06:44:43 6232 acc_Audit: tlhcb008.213:[log in to unmask] grant gsi [log in to unmask] read lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DS
>> T/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:46:42 lcg2160 docker: 200722 06:46:42 4201 XrootdXeq: u41.7:4284@localhost pvt IP46 login as xrootd
>> Jul 22 07:46:45 lcg2160 docker: 200722 06:46:45 4990 XrdLink: Unable to send to u41.7:4307@localhost; broken pipe
>> Jul 22 07:46:45 lcg2160 docker: 200722 06:46:45 4990 XrootdXeq: u41.7:4307@localhost disc 0:03:10 (send failure)
>> Jul 22 07:46:45 lcg2160 docker: ceph_close: closed fd 7407 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 1377, write ops count 0
>> Jul 22 07:46:46 lcg2160 docker: ceph_open: fd 7420 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:47:46 lcg2160 docker: 200722 06:47:46 4900 XrootdXeq: u41.7:4322@localhost pvt IP46 login as xrootd
>> Jul 22 07:47:48 lcg2160 docker: 200722 06:47:48 706 XrdLink: Unable to send to u41.7:4284@localhost; broken pipe
>> Jul 22 07:47:48 lcg2160 docker: 200722 06:47:48 706 XrootdXeq: u41.7:4284@localhost disc 0:01:06 (send failure)
>> Jul 22 07:47:48 lcg2160 docker: ceph_close: closed fd 7420 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 459, write ops count 0
>> Jul 22 07:47:50 lcg2160 docker: ceph_open: fd 7424 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:48:50 lcg2160 docker: 200722 06:48:50 3824 XrootdXeq: u41.7:4298@localhost pvt IP46 login as xrootd
>> Jul 22 07:48:54 lcg2160 docker: 200722 06:48:54 5641 XrdLink: Unable to send to u41.7:4322@localhost; broken pipe
>> Jul 22 07:48:54 lcg2160 docker: 200722 06:48:54 5641 XrootdXeq: u41.7:4322@localhost disc 0:01:08 (send failure)
>> Jul 22 07:48:54 lcg2160 docker: ceph_close: closed fd 7424 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 459, write ops count 0
>> Jul 22 07:49:00 lcg2160 docker: ceph_open: fd 7430 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:50:00 lcg2160 docker: 200722 06:50:00 4201 XrootdXeq: u41.7:4308@localhost pvt IP46 login as xrootd
>> Jul 22 07:50:08 lcg2160 docker: 200722 06:50:08 4811 XrdLink: Unable to send to u41.7:4298@localhost; broken pipe
>> Jul 22 07:50:08 lcg2160 docker: 200722 06:50:08 4811 XrootdXeq: u41.7:4298@localhost disc 0:01:18 (send failure)
>> Jul 22 07:50:08 lcg2160 docker: ceph_close: closed fd 7430 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 459, write ops count 0
>> Jul 22 07:50:10 lcg2160 docker: ceph_open: fd 7434 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:51:10 lcg2160 docker: 200722 06:51:10 4816 XrootdXeq: u41.7:4307@localhost pvt IP46 login as xrootd
>> Jul 22 07:51:14 lcg2160 docker: 200722 06:51:14 4146 XrdLink: Unable to send to u41.7:4308@localhost; broken pipe
>> Jul 22 07:51:14 lcg2160 docker: 200722 06:51:14 4146 XrootdXeq: u41.7:4308@localhost disc 0:01:14 (send failure)
>> Jul 22 07:51:14 lcg2160 docker: ceph_close: closed fd 7434 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 459, write ops count 0
>> Jul 22 07:51:20 lcg2160 docker: ceph_open: fd 7437 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>> Jul 22 07:52:20 lcg2160 docker: 200722 06:52:20 4885 XrootdXeq: u41.7:4315@localhost pvt IP46 login as xrootd
>> Jul 22 07:52:26 lcg2160 docker: 200722 06:52:26 5641 XrdLink: Unable to send to u41.7:4307@localhost; broken pipe
>> Jul 22 07:52:26 lcg2160 docker: 200722 06:52:26 5641 XrootdXeq: u41.7:4307@localhost disc 0:01:16 (send failure)
>> Jul 22 07:52:26 lcg2160 docker: ceph_close: closed fd 7437 for file prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst, read ops count 459, write ops count 0
>> Jul 22 07:52:30 lcg2160 docker: ceph_open: fd 7442 associated to lhcb:prod/lhcb/LHCb/Collision15em/ALLTURBO.DST/00046157/0000/00046157_00001735_1.allturbo.dst
>
> --
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub:
> https://github.com/xrootd/xrootd/issues/1259#issuecomment-663464334
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/xrootd/xrootd/issues/1259#issuecomment-663814663",
"url": "https://github.com/xrootd/xrootd/issues/1259#issuecomment-663814663",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
