 |
 |
I want to confirm that the fix for this issue will be in 5.3.0 which is comming out shortly. On Wed, 30 Jun 2021, Thomas Byrne - STFC UKRI wrote: > Hi Rob, > > Ah, sorry to hear you?ve run into this too. I confirmed it was the read/pgread difference by comparing the f-stream info with the server log output with the ?xrootd.trace fs? directive set. I could see the pgreads and reads hitting the server, and then compared that against the f-stream close info. > > I?ve recently created a ?somewhat rough round the edges? python program to act as an f-stream collector at RAL, loosely based on an older t-stream collector someone else wrote here. It runs a udp server, and parses messages when they come in, and fires them off to elasticsearch in bulk. I struggled a lot with consolidation of the various messages (user conn, file open, file close) to form a meaningful ?file access event? without a) using excessive amount of memory at scale and b) being unable to restart the collector without ending up with ?incomplete? events for some time after. I ended up writing functionality for the collector to query elasticsearch for the previous user connection and file open messages if they aren?t in its memory cache, which is surprisingly performant, and the collector has been working well for us. > > I?m aiming to get it up on our github in the next few weeks (I need to have a serious tidy first to make it generally usable), but if you wanted to look at it for inspiration I?d be happy to send it to you in the meantime, the f-stream parsing stuff is all fairly solid, and might be useful for you to confirm things work how you expected. > > Cheers, > Tom > > From: Robert Currie <[log in to unmask]> > Sent: 30 June 2021 16:44 > To: Byrne, Thomas (STFC,RAL,SC) <[log in to unmask]> > Cc: [log in to unmask] > Subject: Re: pgread stats not recorded in f-stream monitoring XFR/close messages > > > Hi Tom, > > I think I've been seeing similar issues in Edinburgh with our (disk) XCache (XRootD-5.2). > This might explain why I've been struggling to understand some of the fstream data we've been ingesting from the XCache's themselves. > > If I can ask, what collector are you using for ingesting the XRootD metrics?, it sounds like you're doing a similar thing to the ALICE monitoring. > > Best Regards, > > Rob > > On 2021-06-30 16:20, Thomas Byrne - STFC UKRI wrote: > This email was sent to you by someone outside the University. > You should only click on links or attachments if you are certain that the email is genuine and the content is safe. > > Hi all, > > > > After upgrading our external (memory caching) proxies to XRootD 5, we was spotted that some of our internal monitoring is showing less data transferred than expected in some cases. Our monitoring uses the f-stream, and is mostly just consuming the f-stream close messages XFR and OPS sections, along with other messages needed to make sense of them (user connections, fstream opens). > > > > I tracked it down to our XRootD 5 gateways serving some reads as pgreads (specifically, reads from offsite XCaches). I don?t fully understand what pgreads are[1], but I think the change in reporting comes from the fact that previously, our XRootD 4 gateways did not support pgreads, so the pgreads fell back to reads, which then contributed to the XFR ?read? total in the close message. Now the gateways are serving pgreads, and I can?t work out how to get the pgread summary stats. As far as I can tell, pgread operations don?t contribute to any XFR or OPS counters in the f-stream close messages. > > > > Is there a setting I?m missing when specifying the monitoring directive[2], or have I missed a f-stream datagram format change that adds pgread stats? Or alternatively, have I completely misunderstood what is going on here? > > > > Thanks, > > Tom > > > > [1] I think they are some sort of paged read that can make use of consistency checking ?per page? if the server supports it, but I may be completely off the mark here? > > > > [2] We currently use these monitoring directives: > > xrootd.monitor all auth fstat 10s ops lfn xfr 5 ident 5m dest fstat info user redir HOST:PORT > > > > > > This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. > > ________________________________ > > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336. > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1