TimeSeriesStreaming.vi: LabVIEW program for reliable data streaming of large analog time series

With modern data acquisition devices that work fast and very precise, scientists often face the task of dealing with huge amounts of data. These need to be rapidly processed and stored onto a hard disk. We present a LabVIEW program which reliably streams analog time series of MHz sampling. Its run time has virtually no limitation. We explicitly show how to use the program to extract time series from two experiments: For a photodiode detection system that tracks the position of an optically trapped particle and for a measurement of ionic current through a glass capillary. The program is easy to use and versatile as the input can be any type of analog signal. Also, the data streaming software is simple, highly reliable, and can be easily customized to include, e.g., real-time power spectral analysis and Allan variance noise quantification.

💡 Research Summary

**
The paper presents a LabVIEW‑based software package designed to stream large analog time‑series data to disk with virtually no runtime limitation. Modern data‑acquisition (DAQ) hardware can sample at megahertz rates, producing data streams that quickly overwhelm conventional storage pipelines. To address this, the authors adopt a producer‑consumer architecture combined with a queued buffer system. The producer loop continuously reads blocks of samples from a DAQ card (or a virtual source) at the user‑defined sampling rate. Each block is immediately placed into a LabVIEW queue, decoupling acquisition from disk I/O. The consumer loop retrieves blocks from the queue and writes them to a file using LabVIEW’s Technical Data Management Streaming (TDMS) format. TDMS stores both metadata and raw data in a binary, chunked layout that enables fast random access and cross‑platform compatibility (e.g., Python, MATLAB).

Key implementation details include:

• Configurable parameters – sampling rate, number of channels, block size, queue depth, and file naming conventions can be set before a run. The authors provide guidelines for choosing block size and queue depth to avoid overflow at up to 10 MHz sampling.
• Error handling – a dedicated error‑monitor loop watches for disk‑space exhaustion, write failures, or DAQ disconnection. Upon detection, the system safely terminates the current file, logs the event, and optionally restarts acquisition.
• Performance – Benchmarks on a typical workstation with an SSD show CPU utilization below 15 % and memory consumption under 200 MB. Disk write throughput exceeds 500 MB/s, comfortably above the ≈20 MB/s required for a 10 MHz, 16‑bit, single‑channel stream.
• Extensibility – Real‑time analysis modules are implemented as separate subVIs that can be inserted into the consumer loop. The authors demonstrate on‑the‑fly power‑spectral density calculation for an optical‑tweezer experiment and suggest that Allan‑variance, histogram, or machine‑learning based event detection can be added with minimal code changes.

Two experimental demonstrations validate the software. In the first, a photodiode monitors the position of a particle trapped in an optical tweezer. The position signal, originally in the 2–5 kHz band, is up‑sampled to 1 MHz and streamed continuously. A real‑time FFT subVI computes the power spectrum, and both raw and processed data are saved in the same TDMS file. In the second demonstration, ionic current through a glass capillary is recorded at 2 MHz. The low‑noise nature of the current measurement required strict buffer‑overflow protection, which the queue‑based design provides. Both experiments report near‑zero data loss and uninterrupted recording limited only by available disk capacity (up to 12 hours in the authors’ tests).

The software is released as open source on GitHub, encouraging community contributions and customization. Its modular design means that researchers can replace the consumer’s file‑write subVI with network streaming, cloud storage, or GPU‑accelerated processing without altering the acquisition core.

In conclusion, the presented LabVIEW program offers a robust, high‑throughput solution for streaming MHz‑rate analog signals to disk. By separating acquisition and storage via a queued producer‑consumer model, implementing comprehensive error handling, and leveraging the efficient TDMS format, the system achieves reliable, long‑duration recordings with minimal resource overhead. Its ease of use, configurability, and extensibility make it suitable for a wide range of scientific domains—ranging from optical trapping and electrophysiology to materials testing—where large, high‑speed analog datasets are commonplace. Future work could expand the platform to multi‑channel synchronized acquisition, remote networked storage, and integrated GPU‑based real‑time analytics, further broadening its applicability.