 |
 |
Hi again, It sounds like we're doing broadly the same thing in Edinburgh, we've just added some new features to the OSG collector to be able to parse more of the xrootd data than it currently does. Happy to share the code if you're interested in trying to parse the g-stream data and a few other bits of monitoring data dropped by default. Right now we're working on trying to understand some of the metrics we're getting out in more detail to try and see if we're understanding it all correctly. Regards, Rob On 2021-03-25 20:03, Andrew Hanushevsky wrote: > Hi Pail, > > And to add additional Informat, Ilija has also done this. Here is what > he has to say.... > > ------------------ > Yes I did it long time ago. I had a python script that was receiving > summary stream, repackaging it into ES documents and sending them to > indexing. These days I only collect pfc stream. It gets sent to our > logstash and from there to ES. > > If needed I can try to dig out that summary receiving code, I think > CMS people use it even today for stashcache monitoring. > ------------------ > > So, there appear to be two options here. > > Andy > > > On Thu, 25 Mar 2021, Robert Currie wrote: > >> Hi Paul, >> >> We've recently done this in Edinburgh as part of GridPP to support the >> UK XCache efforts (viewable results here: >> https://gridpp.monitoring.edi.scotgrid.ac.uk/app/dashboards#/view/52a44100-75ec-11eb-9e06-fbd4e6c2af73) >> >> We also gave short presentation on this to the UK community here >> https://drive.google.com/drive/folders/1PywvuULoFtWVHTtumX4kajks3a8Ng5bR >> and have submitted a vCHEP'21 abstract/writeup on the work done. >> >> Short answer for how we did it was to locally fork an existing Python3 >> based collector and then to build a message-queue ingestion system >> based on a cluster of docker containers to ingest this into our main >> elasticsearch instance. >> It's not yet perfect, but this works well enough for us to monitor >> both our transparent XRrootD proxy instance (and those at remote >> sites) as well as the XRootD components of our DPM-SE in realtime. >> >> Happy to give some pointers if you're interested on working on >> something like this if we can help avoid duplication of effort. >> >> Rob >> >> On 2021-03-25 08:43, Paul MUSSET wrote: >>> Hello, >>> >>> Has anyone put the XRootD summary monitoring (obtained via xrd.report >>> and mpxstats) into elasticsearch ? If yes, can you tell me how you >>> did >>> it ? >>> >>> Cheers >>> Paul >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> >> -- The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1