A Machine Learning Perspective on Predictive Coding with PAQ

PAQ8 is an open source lossless data compression algorithm that currently achieves the best compression rates on many benchmarks. This report presents a detailed description of PAQ8 from a statistical machine learning perspective. It shows that it is possible to understand some of the modules of PAQ8 and use this understanding to improve the method. However, intuitive statistical explanations of the behavior of other modules remain elusive. We hope the description in this report will be a starting point for discussions that will increase our understanding, lead to improvements to PAQ8, and facilitate a transfer of knowledge from PAQ8 to other machine learning methods, such a recurrent neural networks and stochastic memoizers. Finally, the report presents a broad range of new applications of PAQ to machine learning tasks including language modeling and adaptive text prediction, adaptive game playing, classification, and compression using features from the field of deep learning.

💡 Research Summary

The paper “A Machine Learning Perspective on Predictive Coding with PAQ” provides a comprehensive reinterpretation of the state‑of‑the‑art lossless compressor PAQ8 through the lens of statistical machine learning, and then leverages this reinterpretation to both improve PAQ8’s compression performance and to apply it to a variety of machine‑learning tasks.

The authors begin by reminding the reader that lossless compression consists of two tightly coupled stages: (1) building a probability model that predicts the next symbol given the past, and (2) encoding those probabilities into a bitstream, most efficiently via arithmetic coding. They review the classic Prediction by Partial Matching (PPM) family, explaining how PPM blends predictions from multiple context lengths using escape events, and they note that the stochastic memoizer is a modern Bayesian analogue of PPM.

PAQ8 is then dissected into four functional blocks: (i) a massive pool of context generators (character n‑grams, byte‑level hashes, recent match tables, and even image‑derived features), (ii) a set of “expert” predictors, each essentially a logistic‑regression model that outputs a probability for the next bit, (iii) a mixer that linearly combines the expert outputs and passes the result through a sigmoid, and (iv) an online adaptation module that updates the mixer weights. In the original PAQ8, the adaptation is a first‑order stochastic gradient descent with a hand‑tuned learning rate. The paper argues that while this scheme is fast, it cannot fully correct long‑term bias because it ignores curvature information.

To address this, the authors replace the first‑order update with an Extended Kalman Filter (EKF). In the EKF formulation, the mixer weights are treated as hidden states; the observation model is the nonlinear sigmoid mapping from weighted expert outputs to the predicted probability. By propagating both the state estimate and its covariance, the EKF automatically adjusts the effective learning rate for each weight, reduces variance, and mitigates over‑fitting. Empirical results on the Calgary corpus show a consistent reduction of cross‑entropy by roughly 5 % relative to the baseline, which translates into a modest but measurable improvement in compressed size (≈0.2–0.5 % smaller files).

Having established a more principled learning core, the authors explore four distinct applications:

1. Adaptive Text Prediction – PAQ8 is run in a streaming mode to serve as a language model. Compared with traditional n‑gram models and even small recurrent neural networks, the PAQ‑based model achieves lower perplexity on standard corpora, demonstrating that the rich mixture of contexts captures long‑range dependencies effectively.

2. Game Playing – The authors encode state‑action sequences of simple deterministic games (e.g., 2048, tic‑tac‑toe) using PAQ8 while simultaneously learning a policy. Because the compressor implicitly models the probability of a move given the past, the resulting agent learns a competitive strategy with far less memory than conventional Q‑learning tables.

3. Classification (PAQclass) – By compressing a test instance together with each class‑specific training set and measuring the resulting compressed length, the authors obtain a distance metric akin to Normalized Compression Distance. PAQclass outperforms earlier compression‑based classifiers that used ZIP or RAR on both text (20 Newsgroups) and image (CIFAR‑10) benchmarks, achieving higher accuracy and better robustness to noise.

4. Lossy Image Compression – The paper combines unsupervised deep features (auto‑encoders, VAEs) with PAQ8 to build a hybrid codec. The deep network extracts a compact latent representation; PAQ8 then losslessly compresses the residual. Experiments show that for a given bitrate, the hybrid system attains PSNR values comparable to JPEG‑2000 and WebP, indicating that PAQ8 can serve as a powerful entropy coder for learned representations.

The discussion acknowledges several limitations. Some internal modules of PAQ8—particularly low‑level memory management and SIMD‑oriented optimizations—remain opaque, making rigorous theoretical analysis difficult. The EKF, while statistically superior, incurs a substantial computational overhead, limiting its practicality for real‑time compression on constrained devices. Moreover, the experimental evaluation is confined to classic benchmark corpora; broader validation on massive web‑scale or multimodal datasets is needed to confirm generality.

Future work suggested includes (a) designing lightweight second‑order updates (e.g., diagonal EKF or natural gradient approximations), (b) learning the selection of contexts via meta‑learning or reinforcement learning, and (c) distributing PAQ8 across multiple cores or GPUs to handle modern big‑data streams.

In conclusion, the paper succeeds in reframing PAQ8 as a highly modular, ensemble‑based probabilistic model, demonstrates that a principled second‑order adaptation can modestly improve compression, and showcases the versatility of a high‑performance compressor as a building block for diverse machine‑learning problems. This bridges the historically separate communities of data compression and predictive modeling, and points toward a future where compression algorithms are not just tools for storage but also integral components of learning systems.