Approximate learning of parsimonious Bayesian context trees

Models for categorical sequences typically assume exchangeable or first-order dependent sequence elements. These are common assumptions, for example, in models of computer malware traces and protein sequences. Although such simplifying assumptions lead to computational tractability, these models fail to capture long-range, complex dependence structures that may be harnessed for greater predictive power. To this end, a Bayesian modelling framework is proposed to parsimoniously capture rich dependence structures in categorical sequences, with memory efficiency suitable for real-time processing of data streams. Parsimonious Bayesian context trees are introduced as a form of variable-order Markov model with conjugate prior distributions. The novel framework requires fewer parameters than fixed-order Markov models by dropping redundant dependencies and clustering sequential contexts. Approximate inference on the context tree structure is performed via a computationally efficient model-based agglomerative clustering procedure. The proposed framework is tested on synthetic and real-world data examples, and it outperforms existing sequence models when fitted to real protein sequences and honeypot computer terminal sessions.

💡 Research Summary

The paper introduces Parsimonious Bayesian Context Trees (PBCT), a novel variable‑order Markov modeling framework that combines Bayesian inference with context‑tree representations to capture long‑range dependencies in categorical sequences while dramatically reducing the number of parameters. Traditional fixed‑order Markov models require V^D parameters for a vocabulary of size V and order D, quickly becoming infeasible for realistic vocabularies. PBCT addresses this by allowing each node in the tree to contain a subset of symbols rather than a single symbol, thereby clustering contexts that share the same predictive distribution. The tree is generated recursively: at each non‑leaf node the children form a partition of the current symbol set, and this partition is drawn from a Chinese Restaurant Process (CRP) with concentration parameter α. Smaller α (or depth‑dependent decay of α) yields fewer clusters, controlling model complexity.

Each leaf node is assigned a categorical distribution ϕ_e with a Dirichlet(η) prior. Because of conjugacy, the marginal likelihood of an observed sequence under a given tree T can be computed analytically as a product of multivariate beta functions, p(x|T)=∏_e B(X_e+η)/B(η), where X_e are symbol‑counts conditioned on the leaf’s context. This closed‑form score enables a Bayesian model selection criterion for tree structures.

Exact Bayesian search over all possible trees is combinatorial, so the authors propose an efficient model‑based agglomerative clustering algorithm. Starting from the root, each symbol initially forms its own cluster. At each step the algorithm evaluates the multiplicative increase in marginal likelihood that would result from merging any pair of clusters, s_{i,j}=p(x|T_merged)/p(x|T_current). The pair with the largest increase (if >1) is merged, and the process recurses down the tree until no beneficial merges remain or a predefined maximum depth D is reached, at which point the node becomes a leaf. This greedy scheme runs in O(V²·D) time, far faster than dynamic‑programming approaches for variable‑order Markov models.

Experiments on synthetic data demonstrate that the algorithm reliably recovers the true generating tree, even under moderate noise. Real‑world evaluations on protein sequences (20‑amino‑acid alphabet) and honeypot command‑line logs (several thousand distinct commands) show that PBCT consistently outperforms fixed‑order Markov models (orders 3–5) and previously proposed Bayesian Context Trees. Gains are observed both in log‑likelihood (5–12 % improvement) and predictive accuracy, while the number of free parameters is reduced by an order of magnitude, leading to lower memory consumption and faster inference.

In summary, PBCT offers a scalable Bayesian solution for learning parsimonious, variable‑order context structures in categorical sequences. By clustering contexts via a CRP prior and employing a fast agglomerative inference scheme, it balances expressive power with computational tractability, making it suitable for streaming, bioinformatics, and cybersecurity applications where long‑range dependencies matter but resources are limited. Future work may extend the approach to much larger vocabularies, multi‑sequence joint learning, and non‑categorical data modalities.