 |
 |
Hi Andy, Thanks for the response, I appreciate that there is no ideal solution, but adding it to the read totals seems reasonable to me. Cheers, Tom > -----Original Message----- > From: Andrew Hanushevsky <[log in to unmask]> > Sent: 30 June 2021 23:19 > To: Byrne, Thomas (STFC,RAL,SC) <[log in to unmask]> > Cc: [log in to unmask]; [log in to unmask] > Subject: RE: pgread stats not recorded in f-stream monitoring XFR/close > messages > > Hi Tom and Rob, > > Yes, this is a known problem. The reason that it occurs is because we could not > agree on a backwardly compatible way to report that number. I suppose we can > simply add it to the regular read number but that would upset another set of > people. There is no good solution (i.e. one that doesn't elicit complaints). I just > bite the bullet and add it to the total. > > Andy > > > On Wed, 30 Jun 2021, Thomas Byrne - STFC UKRI wrote: > > > Hi Rob, > > > > Ah, sorry to hear you?ve run into this too. I confirmed it was the read/pgread > difference by comparing the f-stream info with the server log output with the > ?xrootd.trace fs? directive set. I could see the pgreads and reads hitting the > server, and then compared that against the f-stream close info. > > > > I?ve recently created a ?somewhat rough round the edges? python program to > act as an f-stream collector at RAL, loosely based on an older t-stream collector > someone else wrote here. It runs a udp server, and parses messages when they > come in, and fires them off to elasticsearch in bulk. I struggled a lot with > consolidation of the various messages (user conn, file open, file close) to form a > meaningful ?file access event? without a) using excessive amount of memory at > scale and b) being unable to restart the collector without ending up with > ?incomplete? events for some time after. I ended up writing functionality for the > collector to query elasticsearch for the previous user connection and file open > messages if they aren?t in its memory cache, which is surprisingly performant, > and the collector has been working well for us. > > > > I?m aiming to get it up on our github in the next few weeks (I need to have a > serious tidy first to make it generally usable), but if you wanted to look at it for > inspiration I?d be happy to send it to you in the meantime, the f-stream parsing > stuff is all fairly solid, and might be useful for you to confirm things work how you > expected. > > > > Cheers, > > Tom > > > > From: Robert Currie <[log in to unmask]> > > Sent: 30 June 2021 16:44 > > To: Byrne, Thomas (STFC,RAL,SC) <[log in to unmask]> > > Cc: [log in to unmask] > > Subject: Re: pgread stats not recorded in f-stream monitoring XFR/close > messages > > > > > > Hi Tom, > > > > I think I've been seeing similar issues in Edinburgh with our (disk) XCache > (XRootD-5.2). > > This might explain why I've been struggling to understand some of the fstream > data we've been ingesting from the XCache's themselves. > > > > If I can ask, what collector are you using for ingesting the XRootD metrics?, it > sounds like you're doing a similar thing to the ALICE monitoring. > > > > Best Regards, > > > > Rob > > > > On 2021-06-30 16:20, Thomas Byrne - STFC UKRI wrote: > > This email was sent to you by someone outside the University. > > You should only click on links or attachments if you are certain that the email is > genuine and the content is safe. > > > > Hi all, > > > > > > > > After upgrading our external (memory caching) proxies to XRootD 5, we was > spotted that some of our internal monitoring is showing less data transferred > than expected in some cases. Our monitoring uses the f-stream, and is mostly > just consuming the f-stream close messages XFR and OPS sections, along with > other messages needed to make sense of them (user connections, fstream > opens). > > > > > > > > I tracked it down to our XRootD 5 gateways serving some reads as pgreads > (specifically, reads from offsite XCaches). I don?t fully understand what pgreads > are[1], but I think the change in reporting comes from the fact that previously, > our XRootD 4 gateways did not support pgreads, so the pgreads fell back to > reads, which then contributed to the XFR ?read? total in the close message. Now > the gateways are serving pgreads, and I can?t work out how to get the pgread > summary stats. As far as I can tell, pgread operations don?t contribute to any > XFR or OPS counters in the f-stream close messages. > > > > > > > > Is there a setting I?m missing when specifying the monitoring directive[2], or > have I missed a f-stream datagram format change that adds pgread stats? Or > alternatively, have I completely misunderstood what is going on here? > > > > > > > > Thanks, > > > > Tom > > > > > > > > [1] I think they are some sort of paged read that can make use of consistency > checking ?per page? if the server supports it, but I may be completely off the > mark here? > > > > > > > > [2] We currently use these monitoring directives: > > > > xrootd.monitor all auth fstat 10s ops lfn xfr 5 ident 5m dest fstat info user redir > HOST:PORT > > > > > > > > > > > > This email and any attachments are intended solely for the use of the named > recipients. If you are not the intended recipient you must not use, disclose, copy > or distribute this email or any of its attachments and should notify the sender > immediately and delete this email from your system. UK Research and Innovation > (UKRI) has taken every reasonable precaution to minimise risk of this email or any > attachments containing viruses or malware but the recipient should carry out its > own virus and malware checks before opening the attachments. UKRI does not > accept any liability for any losses or damages which the recipient may sustain due > to presence of any viruses. Opinions, conclusions or other information in this > message and attachments that are not related directly to UKRI business are > solely those of the author and do not represent the views of UKRI. > > > > ________________________________ > > > > Use REPLY-ALL to reply to list > > > > To unsubscribe from the XROOTD-L list, click the following link: > > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > > > > > The University of Edinburgh is a charitable body, registered in Scotland, with > registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh > Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336. > > > > > ################################################################# > ####### > > Use REPLY-ALL to reply to list > > > > To unsubscribe from the XROOTD-L list, click the following link: > > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI. ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1