Data Compression with Stochastic Codes

Machine learning has had a major impact on data compression over the last decade and inspired many new, exciting theoretical and applied questions. This paper describes one such direction – relative entropy coding – which focuses on constructing stochastic codes, primarily as an alternative to quantisation and entropy coding in lossy source coding. Our primary aim is to provide a broad overview of the topic, with an emphasis on the computational and practical aspects currently missing from the literature. Our goal is threefold: for the curious reader, we aim to provide an intuitive picture of the field and convince them that relative entropy coding is a simple yet exciting emerging field in data compression research. For a reader interested in applied research on lossy data compression, we provide an account of the most salient contemporary applications. Finally, for the reader who has heard of relative entropy coding but has never been quite sure what it is or how the algorithms fit together, we hope to illustrate how simple and elegant the underlying constructions are.

💡 Research Summary

This paper surveys the emerging paradigm of relative entropy coding (REC), positioning it as a stochastic alternative to traditional quantisation and entropy coding in lossy source compression. The authors begin with a historical analogy to the “reverse Cardan” grille—a steganographic technique where a shared random text (or time‑series) serves as a common randomness source. By interpreting the distances between grille holes as an integer sequence, they show that, for a memoryless source, the average per‑symbol code length matches the Shannon entropy, thereby establishing a bridge between classic source coding and the new stochastic framework.

Formal definitions follow: a stochastic code is a triple (Z, enc_Z, dec_Z) where Z is a random variable independent of the source X, enc_Z maps X to a binary string, and dec_Z, using Z and the binary string, produces a sample Y distributed according to a prescribed conditional distribution P_{Y|X}. A relative entropy code further requires that the expected code length be within a constant (plus a logarithmic term) of the mutual information I