 |
 |
Hi K.T., Thansk for your answer, here's one last remark below. On 06/01/2015 04:05 PM, Kian-Tat Lim wrote: > Fabrice and the Qserv team: > >> - Standard de facto is ELK (ElasticSearch, Logstash, Kibana), which >> is a splunk-like open-source software. > Yes. > >> But in2p3 sysadmin recommends >> to use syslog-ng instead of LogStash, because it is ten times faster >> and it can support high load. > That is interesting information. > >> - Good practive is to produce structured log (i.e. JSON format for >> example). > I'm not sure about this yet. > > But whatever Qserv uses must be compatible with what the rest of > the operational system, including the Alert, Data Release, and > Calibration Products Productions uses. Qserv cannot make these > decisions for itself, unfortunately. You need to discuss with SQuaRE > and NCSA. We can have 2 appenders here, one for flat files and one for structured data, IN2P3 ELK/syslog-ng expert emphasize this point as a very string requirement to ease initialization/maintenance of monitoring tool. Indeed, regexp matching seems to very cost-effective/fragile when managing multiple apps. Note that most of the keys can be easily created (data, severity, logger, pid, message, ...) Nevertheless we can add optional keys in the message if we want to track performance for a given query (i.e. have keys like query id, query execution step, time, ... instead of a plain-text message). >> Task to do: >> >> - add a JSON appender to Qserv(/xrootd?) logger >> - install/configure a virtual machine ccqserv-mon with ElasticSearch >> and Kibana, >> - install/configure syslog-ng on all cc-in2p3 Qserv cluster > None of this should happen before those discussions occur. > >> Shall we create new tickets in epic DM-1823 (Design and implement >> monitoring tool for Qserv)? > Please don't, or at least make them dependent on a preceding > design and discussion ticket. > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1