 |
 |
Hi Julia, The way we scaled xrootd monitoring to avoid burdening the server was to simply shift the workload to the collector. That meant that state information was reported "just in time" at the point it was available. Much of that state information is not kept beyond the reporting time interval. The information that is kept is not easily accessible, if it's accessible at all, by the time monitoring occurs. What we provide is a simple reference to previosuly reported data which avoids retransmission of that data and keeps the amount of monitoring data sent over the wire in balance with the amount of data that we provide to the client. Internal monitoring API's and structures are architected to fit into that scheme. This was not short-sighted. It allowed us to provide far more extensive information at a very low cost accomodating divergent and evolving needs without having to revisit the server implementation which, frankly, is a very expensive proposition. Yes, that all works as long as you don't loose too many UDP packets. That is why the reliable conversion of UDP to MQ is the long term solution and will address the specific problem at hand. You will be able to get exactly the information you need once we can stop loosing packets. I ask for your patience while we implement what will be the satisfactory solution for the whole community. Andy Andy On Fri, 25 Jun 2021, Julia Andreeva wrote: > Hi Andrew, > > Thank you for the prompt answer. > Yes, we know about the development of the local UDP collector converting data into MQ stream. > This is certainly a good direction. > > However, this does not simplify the task of processing a lot of reports related to the same operation further in the chain, which has impact on the collector scalability. And we were wondering if we can substantially decrease amount of data to be processed if file close report contains just few more attributes. > If I understand correctly your argument about serious architectural changes, it means that xrootd server does not keep a track of complete info related to a given operation during life time of this operation. Is it correct? For example, the owner of a given operation should be preserved during lifetime of operation, shouldn't it? > > Thank you > > Cheers > Julia > ________________________________________ > From: [log in to unmask] [[log in to unmask]] on behalf of Andrew Hanushevsky [[log in to unmask]] > Sent: 25 June 2021 09:40 > To: Julia Andreeva > Cc: [log in to unmask]; Alessandra Forti; Maarten Litmaath > Subject: Re: xrootd monitoring request > > Hi Julia, > > The problem is that it would require significant architectutal changes, > not to mention a whole new set of documentation. We are taking a different > track to address this problem that we think is a more general solution > that should address a wider range of needs. > > We already have two collectors that know how to piece the information > together to essentially provide what you are requesting. The problem is > that if one of the required UDP packets is lost it makes it impossible > to provide complete information. The plan is to provide a local UDP > collector where the likelihood of packet loss is minimal and convert > the stream to a MQ stream (e.g. rabbitMQ). That would minimize > information loss. We have one prototype and our collaborators are working > on some others. > > I don't have an exact timeline but given your request I will try to > establish one that is hopefully timely. This particular effort is > community supported so we need to take that into account. However, we do > have OSG and two other collaborative partners participating so it > shouldn't take too long to provide a solution. I hope that this addresses > your request. > > Andy > > > > On Thu, 24 Jun 2021, Julia Andreeva wrote: > >> Dear colleagues, >> >> We are discussing various options to improve the situation with xrootd monitoring on the WLCG infrastructure. >> One of the options we consider is to try ALICE approach which consists of processing only file close reports. This looks to be efficient regarding optimization of the amount of data volume to be processed by the collector. However, ALICE uses dedicated xrootd servers, while for monitoring of the xrootd traffic on the WLCG infrastructure, we need to know the owner of the operation in terms of the VO. >> >> Two other things which would be useful to have in the file close report is the start time stamp of the operation and application level meta info which is now possible to provide with the monitoring reports, but if I am not wrong , it is send in the beginning. >> >> So our question is, would it be possible to extend the file close report with information about: VO, start time stamp of the operation and application level meta info. >> >> Please, let us know. >> >> Thank you >> >> Kind regards >> Julia >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1