A Method for Compressing Parameters in Bayesian Models with Application to Logistic Sequence Prediction Models

Bayesian classification and regression with high order interactions is largely infeasible because Markov chain Monte Carlo (MCMC) would need to be applied with a great many parameters, whose number increases rapidly with the order. In this paper we show how to make it feasible by effectively reducing the number of parameters, exploiting the fact that many interactions have the same values for all training cases. Our method uses a single ``compressed’’ parameter to represent the sum of all parameters associated with a set of patterns that have the same value for all training cases. Using symmetric stable distributions as the priors of the original parameters, we can easily find the priors of these compressed parameters. We therefore need to deal only with a much smaller number of compressed parameters when training the model with MCMC. The number of compressed parameters may have converged before considering the highest possible order. After training the model, we can split these compressed parameters into the original ones as needed to make predictions for test cases. We show in detail how to compress parameters for logistic sequence prediction models. Experiments on both simulated and real data demonstrate that a huge number of parameters can indeed be reduced by our compression method.

💡 Research Summary

The paper tackles a fundamental scalability problem in Bayesian classification and regression models that incorporate high‑order interactions. As the interaction order grows, the number of regression coefficients explodes combinatorially, making Markov chain Monte Carlo (MCMC) inference practically impossible because of prohibitive memory consumption, slow mixing, and difficulty in assessing convergence.
The authors observe that, for a given training set, many interaction patterns (e.g., specific combinations of predictor values) take exactly the same value across all training cases. When this occurs, the individual coefficients attached to those patterns always appear together in the likelihood and therefore only their sum matters for the posterior. The key idea is to replace each such group of coefficients by a single “compressed” parameter that represents the sum of the original parameters.
To keep the Bayesian framework coherent, the paper assumes that the original coefficients have independent symmetric α‑stable priors (e.g., Gaussian for α = 2, Cauchy for α = 1). A crucial property of stable distributions is that the sum of independent α‑stable variables is again α‑stable, with a scale parameter that can be computed analytically. Consequently, the prior distribution of each compressed parameter is known in closed form, and no additional approximation is required.
The compression procedure proceeds in two stages. First, all possible interaction patterns up to a chosen maximal order are organized in a tree structure. For each node (pattern) the algorithm checks whether the pattern’s indicator vector is identical for every training observation. If it is, the node is placed into a “pattern set” together with any other nodes sharing the same indicator vector. Second, each pattern set is assigned one compressed parameter θ̂, defined as the sum of the original coefficients belonging to the set. During MCMC, only the θ̂’s are sampled, dramatically reducing the dimensionality of the state space. After the sampling phase, the compressed parameters can be split back into the original coefficients as needed for prediction; the split is trivial because it merely distributes θ̂ uniformly (or according to a chosen weighting) among the members of its set.
The authors illustrate the method with logistic sequence‑prediction models. In such models the response is binary and the predictors are binary lag variables; the logistic link uses a linear combination of indicator functions for all patterns of length ≤ L. Without compression, the number of coefficients grows as 2^L − 1, quickly reaching millions for moderate L. By applying the compression algorithm, the number of effective parameters stops growing after a relatively low order because many high‑order patterns are indistinguishable on the training data.
Empirical evaluation is performed on both synthetic data (where high‑order interactions are deliberately planted) and real‑world text data (character‑level binary encoding of English Wikipedia sentences). The experiments compare (a) the raw number of parameters, (b) the number after compression, (c) MCMC efficiency metrics such as effective sample size and autocorrelation time, and (d) predictive performance measured by ROC‑AUC and log‑likelihood. Results show reductions of up to 99.9 % in the number of parameters, while predictive accuracy remains essentially unchanged or even improves slightly due to reduced over‑fitting. Moreover, the effective sample size per unit time increases dramatically, confirming that the compressed model mixes far more rapidly.
The paper also discusses limitations. Compression relies on exact pattern identity across the training set; in highly sparse or noisy data the proportion of compressible patterns may be small, limiting the benefit. The current theory is restricted to symmetric α‑stable priors; extending the approach to asymmetric or hierarchical priors would require additional work. The authors suggest future directions such as approximate compression via clustering of similar patterns, and application to more complex Bayesian deep models.
In summary, the work provides a mathematically sound and practically effective technique for reducing the parameter space of Bayesian models with high‑order interactions. By exploiting the closure property of stable distributions, it preserves the original prior structure while enabling scalable MCMC inference. The method is demonstrated on logistic sequence prediction but is applicable to a broad class of Bayesian regression and classification problems where many interaction terms are redundant on the observed data. This contribution opens the door to feasible Bayesian modeling of richly interacting features that were previously out of reach due to computational constraints.