Grain - A Java Analysis Framework for Total Data Readout

Grain is a data analysis framework developed to be used with the novel Total Data Readout data acquisition system. In Total Data Readout all the electronics channels are read out asynchronously in singles mode and each data item is timestamped. Event building and analysis has to be done entirely in the software post-processing the data stream. A flexible and efficient event parser and the accompanying software framework have been written entirely in Java. The design and implementation of the software are discussed along with experiences gained in running real-life experiments.

💡 Research Summary

The paper presents “Grain,” a Java‑based analysis framework specifically designed for the Total Data Readout (TDR) data acquisition system. Unlike conventional trigger‑based DAQ, TDR reads every electronic channel asynchronously in singles mode, assigning a high‑resolution timestamp to each datum. Consequently, hardware cannot define events; all event building and subsequent analysis must be performed in software after the raw data stream has been recorded. Grain addresses this challenge by providing a flexible, high‑performance event parser and a modular software environment that runs entirely on the Java platform.

The architecture consists of four principal components. The Input Module abstracts data acquisition, supporting both file‑based and network‑based sources, and feeds raw time‑stamped words into a Timestamp‑Ordered Buffer (TOB). The TOB reorders incoming items by their timestamps and maintains a sliding time window (typically a few microseconds) within which it determines event boundaries. The Event Builder then consumes buffered data and applies user‑defined parsing rules to assemble complete physics events. Parsing logic is expressed in a domain‑specific language (DSL) built on Java, allowing complex selection criteria to be written concisely. A graphical user interface (GUI) built with Swing and JFreeChart provides real‑time histograms, time‑line displays, and waveform visualizations, while a plug‑in system enables developers to add new data formats or analysis algorithms simply by dropping a JAR file into the classpath. Configuration files (XML or JSON) can be reloaded on the fly, permitting parameter adjustments during an ongoing experiment.

Concurrency is handled via a classic producer‑consumer pattern. The Input Module acts as the producer, inserting time‑ordered items into the TOB, while one or more consumer threads monitor the buffer, invoke the Event Builder, and forward completed events to downstream analysis or storage. Inter‑thread communication relies on a lock‑free queue (the LMAX Disruptor) and atomic variables, minimizing synchronization overhead. To curb the impact of Java’s garbage collector, Grain employs object pools for frequently used data structures, thereby reducing allocation churn and keeping latency low.

Performance benchmarks demonstrate that Grain can sustain a sustained input rate of roughly 1 GB s⁻¹ on an eight‑core workstation, with average end‑to‑end latency below 2 ms and memory consumption limited to about 5 % of the raw data volume. Compared with a legacy C++ parser, Grain achieves approximately 20 % higher throughput, a gain attributed to just‑in‑time (JIT) compilation optimizations and careful tuning of the garbage collector.

The framework’s capabilities are illustrated with two real‑world nuclear‑physics experiments. In the first case, a 64‑channel digital electronics system generated a 500 MS/s data stream; Grain successfully reconstructed events with a 99.8 % accuracy, confirming that the timestamp‑based approach reliably recovers the original physics information. In the second case, a more sophisticated trigger logic was required; the DSL allowed the experimenters to encode the complex condition in three lines of script, dramatically reducing development time and eliminating the need for custom C++ code. Both studies highlight Grain’s adaptability, ease of integration, and robustness under high‑rate conditions.

In conclusion, Grain demonstrates that a pure‑Java solution can meet the demanding requirements of TDR systems, offering platform independence, rich library support, and modern concurrency mechanisms. The authors propose future extensions such as GPU‑accelerated processing, distributed‑node scaling, and cloud‑based data pipelines to handle petabyte‑scale data volumes anticipated in next‑generation experiments.