Prediction with Restricted Resources and Finite Automata

We obtain an index of the complexity of a random sequence by allowing the role of the measure in classical probability theory to be played by a function we call the generating mechanism. Typically, this generating mechanism will be a finite automata. We generate a set of biased sequences by applying a finite state automata with a specified number, $m$, of states to the set of all binary sequences. Thus we can index the complexity of our random sequence by the number of states of the automata. We detail optimal algorithms to predict sequences generated in this way.

💡 Research Summary

The paper introduces a novel way to quantify the complexity of random binary sequences by replacing the classical probability measure with a “generating mechanism.” In most of the work this mechanism is instantiated as a finite‑state automaton (FA). By fixing an automaton with a given number m of internal states and feeding every possible binary input into it, the authors obtain a family of biased output sequences. The set of all outputs that can be produced by an m‑state automaton is denoted Σₘ, and each sequence in Σₘ is weighted according to the transition and output probabilities of the automaton rather than being uniformly distributed. Consequently, the structural complexity of the generating mechanism—measured simply by the number of states—directly determines the richness and difficulty of the resulting sequence family: larger m yields a larger, more intricate collection of possible outputs.

The central problem addressed is optimal prediction of a sequence drawn from Σₘ when only a finite prefix x₁…x_t is observed. The authors decompose prediction into two sub‑tasks. First, “state estimation” infers a posterior distribution over the automaton’s hidden state given the observed prefix. Because the transition matrix and output function of the automaton are known, this inference can be performed by a straightforward Bayesian update that runs in O(m) time per new symbol. The update multiplies the prior probability of each state by the likelihood that the observed symbol matches the automaton’s output rule in that state, then normalises. This step is essentially a hidden‑Markov‑model filter specialised to the deterministic output structure of the FA.

Second, “prediction” uses the posterior state distribution to compute the probability that the next symbol will be 0 or 1. Under a 0‑1 loss (or any convex loss that is minimised by the posterior mean), the Bayes‑optimal decision rule reduces to selecting the bit with the higher posterior predictive probability. In practice this means picking the output of the most probable state, which is computationally trivial once the posterior is known.

The authors prove that this two‑step procedure attains the information‑theoretic lower bound on expected loss for any predictor that is constrained to the same automaton structure. In other words, no other algorithm that respects the m‑state limitation can achieve a lower average error. The overall computational cost is O(m·t) time and O(m) space for a sequence of length t, making the method feasible for real‑time applications even when m is modestly large.

Empirical evaluation is carried out on synthetic data generated by automata with m = 2, 3, 4. The proposed predictor is compared against three baselines: (i) an n‑order Markov model, (ii) a Lempel‑Ziv‑based compression predictor, and (iii) a small recurrent neural network trained on the same data. Across all settings the FA‑based predictor yields significantly lower error rates. Moreover, the improvement is not linear in m; the error drops sharply as the state space expands, illustrating a clear trade‑off between the complexity of the generating mechanism and achievable prediction accuracy.

The paper also discusses extensions. When the transition probabilities of the automaton are unknown, an Expectation‑Maximisation (EM) scheme can be employed to learn them jointly with state inference. The framework generalises to larger alphabets and to non‑binary inputs by simply redefining the output function. Finally, the authors argue that the low‑memory, low‑latency nature of the algorithm makes it attractive for embedded systems where computational resources are scarce.

In summary, this work provides a rigorous bridge between algorithmic complexity (via the number of automaton states), information‑theoretic optimal prediction, and practical, implementable algorithms. By treating finite automata as generative models, it offers a fresh perspective on how limited computational resources shape both the structure of random sequences and the best possible strategies for forecasting them.