Estimating the entropy of binary time series: Methodology, some theory and a simulation study

Partly motivated by entropy-estimation problems in neuroscience, we present a detailed and extensive comparison between some of the most popular and effective entropy estimation methods used in practice: The plug-in method, four different estimators based on the Lempel-Ziv (LZ) family of data compression algorithms, an estimator based on the Context-Tree Weighting (CTW) method, and the renewal entropy estimator. **Methodology. Three new entropy estimators are introduced. For two of the four LZ-based estimators, a bootstrap procedure is described for evaluating their standard error, and a practical rule of thumb is heuristically derived for selecting the values of their parameters. ** Theory. We prove that, unlike their earlier versions, the two new LZ-based estimators are consistent for every finite-valued, stationary and ergodic process. An effective method is derived for the accurate approximation of the entropy rate of a finite-state HMM with known distribution. Heuristic calculations are presented and approximate formulas are derived for evaluating the bias and the standard error of each estimator. ** Simulation. All estimators are applied to a wide range of data generated by numerous different processes with varying degrees of dependence and memory. Some conclusions drawn from these experiments include: (i) For all estimators considered, the main source of error is the bias. (ii) The CTW method is repeatedly and consistently seen to provide the most accurate results. (iii) The performance of the LZ-based estimators is often comparable to that of the plug-in method. (iv) The main drawback of the plug-in method is its computational inefficiency.

💡 Research Summary

The paper addresses the problem of estimating the entropy rate of binary time‑series, a task that arises frequently in neuroscience when quantifying the information content of spike‑train recordings. The authors conduct a comprehensive comparative study of the most widely used entropy estimators and introduce three novel variants. The methods examined are: (1) the classic plug‑in estimator, which directly substitutes empirical frequencies into the entropy formula; (2) four estimators derived from the Lempel‑Ziv (LZ) family of universal data‑compression algorithms; (3) an estimator based on the Context‑Tree Weighting (CTW) algorithm; and (4) a renewal‑process estimator that exploits inter‑event interval statistics.

In the methodology section the authors present three new estimators. Two of them are refined LZ‑based estimators (one LZ78‑style, one LZ77‑style) for which they develop a bootstrap procedure that yields an empirical standard error for each estimate. They also propose a practical “rule‑of‑thumb” for selecting the internal parameters (window length, minimum match length, etc.) based on the sample size. The third new estimator is a renewal‑entropy estimator that is particularly suited to data generated by point processes with independent inter‑arrival times.

The theoretical contributions are substantial. The authors prove that the two new LZ‑based estimators are consistent for any finite‑valued, stationary, ergodic process—a significant strengthening of earlier results that required restrictive Markov‑order assumptions. They also derive an accurate method for approximating the entropy rate of a finite‑state hidden Markov model (HMM) when the model parameters are known. For each estimator, heuristic calculations lead to closed‑form approximations of bias and variance. In particular, the bias of the LZ estimators scales as O(1/ log n) and the bootstrap‑based standard error scales as O(1/√n). By contrast, the plug‑in estimator’s bias behaves like O(2^k / n) where k is the block length, and its computational complexity grows exponentially with k, making it impractical for large k.

The simulation study is extensive. Synthetic binary sequences are generated from a variety of stochastic models: i.i.d. Bernoulli processes, first‑order and higher‑order Markov chains, finite‑state HMMs, and renewal processes with long‑range dependence. Sample lengths range from 10⁴ to 10⁶ symbols, allowing the authors to explore both small‑sample and asymptotic regimes. For each combination of model and estimator, the authors compute the mean‑squared error (MSE), decompose it into bias and variance components, and evaluate computational time and memory usage. The key empirical findings are:

1. Bias dominates error – across all estimators, the bias accounts for roughly 70–90 % of the total MSE, confirming the theoretical expectation that reducing bias is the primary route to improved accuracy.
2. CTW is the most accurate – the Context‑Tree Weighting estimator consistently yields the lowest MSE, especially for higher‑order Markov and HMM data where it outperforms the plug‑in method by 30–50 %. Its adaptive weighting of context trees effectively mitigates bias while maintaining low variance.
3. LZ‑based estimators are competitive – the refined LZ estimators perform on par with, and sometimes slightly better than, the plug‑in estimator, particularly when the sample size exceeds 10⁵. Their bootstrap‑derived standard errors provide useful confidence intervals without substantial additional computation.
4. Plug‑in suffers from computational inefficiency – the need to enumerate all possible blocks of length k leads to exponential growth in both runtime and memory consumption, rendering the method impractical for moderate to large k or long sequences.
5. Renewal estimator is model‑specific – it excels when the data truly follow a renewal process with independent inter‑arrival times, but its performance degrades for general Markovian or HMM‑generated data.

Based on these results, the authors propose practical guidelines: use CTW as the default estimator when accuracy is paramount; adopt the LZ‑based estimators with bootstrap when computational resources are limited or when real‑time estimation is required; resort to the renewal estimator only when the underlying process is known to be renewal‑type; and avoid the plug‑in estimator for anything beyond very short sequences or low‑order Markov models.

Overall, the paper makes a significant contribution by (i) extending the theoretical foundation of LZ‑based entropy estimation, (ii) providing concrete, data‑driven methods for bias and variance assessment, and (iii) delivering a thorough empirical benchmark that clarifies the trade‑offs among the leading entropy‑rate estimators. These insights are directly applicable not only to neuroscience but also to any field that analyzes binary symbolic streams, such as communications, bioinformatics, and quantitative finance.