Hello,

There has been much discussion lately on monitoring tools for the upcoming run within various meetings and on Slack. I thought I would provide a quick status report on where we stand with the various tools, as well as some information on tasks which may need to be completed before the run.

The new suite of online reconstruction tools under development for the past few years has been merged into the hps-java master and currently functions with the 2016 data used as test data. There should not be any significant roadblocks in using this system with the 2021 data.

The new set of tools includes the following components:

• Server for launching instances of the reconstruction
• Client for communicating with the server (e.g. to create/start/stop recon stations)
• Reconstruction stations which book and fill their respective histogram trees
• Aggregator which reads the station histograms and adds them into a combined tree
• Java web application for viewing the individual and combined trees in a web browser

In addition to the web application, the full plot tree can also be loaded and viewed within JAS3, where customization can be performed such as setting axis ranges, laying out plots in tabs or windows, plot styling, etc.

This component-based system has the following advantages over our existing monitoring application:

• Multiple instances of the reconstruction can be run in parallel with the results combined into aggregate histograms.
• Identical monitoring output in the form of histograms can be viewed by multiple shifters or collaboration members simultaneously.
• The web application could be configured so that it is accessible outside of the Hall B network from a URL under our existing hpsweb sub-domain (covered below).
• Shifters need not configure individual monitoring stations to view plots from the reconstruction.
• The reconstruction and booking/filling of histograms is separated from the plotting so that monitoring drivers do not need to perform any styling or layout.

To deploy and use these tools for the run, the appropriate software packages will need to be installed and configured in Hall B, including Java, hps-java along with its set of wrapper scripts, the Tomcat web server, and the web application. There will need to be a work area on disk for storing the configuration files and scripts for this system. Someone will need to oversee this setup and make sure that everything works correctly. I can either do this myself via an SSH connection to Hall B or provide guidance to someone else who will do it. I prefer the latter so that there is an in-house expert who understands how the system works and can perform some amount of debugging and setup themselves on-site. Additionally, certain procedures will need to be established in a playbook for configuring and running the new tools before and after a run, as well as for debugging typical problems. 

Should we decide to make the web application accessible through a public URL, we will need to work with JLAB IT to open up traffic from a Hall B computer on the applicable network ports to our web server. We can open a CCPR in order to start this process, but we will need to identify a host before doing this so it can be added to the outbound white list, in coordination with the DAQ experts who may also need one or more dedicated hosts for the run. I anticipate there will be some back-and-forth on this as we figure out exactly which ports need to be open for the application to function properly. Alternatively, a browser could be opened in VNC or via X11-tunneling to view the plots remotely, though this is more difficult in terms of configuration and likely to incur some amount of lag for users. Should the application be accessible via a public URL, we will use CUE authentication to restrict access to those with an active JLAB computing account within the HPS group, as we already do with our other web applications. 

Matt Graham has reported that the Java-based monitoring application still functions properly based on recent testing. We anticipate no major difficulties in using it for the run. He and I agreed that it is still a useful application and that both systems could be deployed for different purposes. The monitoring application, along with its set of drivers, implements several custom plot layouts, such as for viewing SVT occupancies, which would require considerable work to replicate in the web application. It also has several panels providing additional functionality such as alarms and run statistics, which do not exist in the web application, and, again, would require a considerable amount of work to implement. The new tools could be used for different purposes than the monitoring application, such as real-time DQM running in parallel across many reconstruction stations. 

We are planning on writing the lots from the new online recon system and placing them in the existing drop directory for the DQM which was used for the last major run. These are viewable in a JSROOT application which is already setup and functioning. The writing and copying of the plots may require some additional work, depending on the level of automation which is desired for this procedure. This procedure will allow us to archive and view sets of plots from the run as they become available.

I can devote some amount of time and effort to improvements and bug fixes for the new tools up to and through the run. Patches and updates to the monitoring application can also be made, as necessary, though this application has been stable for some time, and major changes to its code are not foreseen, aside from possible updates to the monitoring drivers themselves.

If you have any questions, comments, or concerns in this area, or would like to bring up any related topics, then please feel free to either reply to this email via the hps-software list or ask in the #software_coordination channel on our Slack.

Thank you!

 

--Jeremy McCormick, SLAC