Common information revisited

One of the main notions of information theory is the notion of mutual information in two messages (two random variables in Shannon information theory or two binary strings in algorithmic information theory). The mutual information in $x$ and $y$ measures how much the transmission of $x$ can be simplified if both the sender and the recipient know $y$ in advance. G'acs and K"orner gave an example where mutual information cannot be presented as common information (a third message easily extractable from both $x$ and $y$). Then this question was studied in the framework of algorithmic information theory by An. Muchnik and A. Romashchenko who found many other examples of this type. K. Makarychev and Yu. Makarychev found a new proof of G'acs–K"orner results by means of conditionally independent random variables. The question about the difference between mutual and common information can be studied quantitatively: for a given $x$ and $y$ we look for three messages $a$, $b$, $c$ such that $a$ and $c$ are enough to reconstruct $x$, while $b$ and $c$ are enough to reconstruct $y$. In this paper: We state and prove (using hypercontractivity of product spaces) a quantitative version of G'acs–K"orner theorem; We study the tradeoff between $\abs{a}, \abs{b}, \abs{c}$ for a random pair $(x, y)$ such that Hamming distance between $x$ and $y$ is $\eps n$ (our bounds are almost tight); We construct “the worst possible” distribution on $(x, y)$ in terms of the tradeoff between $\abs{a}, \abs{b}, \abs{c}$.

💡 Research Summary

The paper revisits the classical notion of common information introduced by Gács and Körner, which captures the part of two correlated sources that can be extracted as a single “common” message. While mutual information quantifies how much knowledge of one source reduces the description length of the other, common information asks whether there exists a third message that can be recovered from both sources with negligible overhead. Gács‑Körner showed that for many joint distributions the common information is strictly smaller than the mutual information; subsequent works (Muchnik–Romashchenko, Makarychev brothers) extended the phenomenon to algorithmic information theory.

The authors introduce a unifying combinatorial framework called the profile of a bipartite graph (E\subseteq X\times Y). A triple ((\alpha,\beta,\gamma)) belongs to the profile if there exist encoders (f,g) such that for every edge ((x,y)\in E) one can find strings (a\in{0,1}^\alpha), (b\in{0,1}^\beta), (c\in{0,1}^\gamma) with (x=f(a,c)) and (y=g(b,c)). In the communication picture, (a) and (b) are private messages sent to two nodes, while (c) is a common broadcast; the lengths (|a|,|b|,|c|) correspond to (\alpha,\beta,\gamma). This formulation captures the trade‑off between private and common communication needed to reconstruct both sources.

Three elementary necessary conditions for a triple to belong to the profile are proved (Proposition 3): (i) (\alpha+\gamma\ge\log|X|) and (\beta+\gamma\ge\log|Y|); (ii) (\alpha+\beta+\gamma\ge\log|E|); (iii) the converse sufficient conditions when the sum of two parameters exceeds the logarithm of the edge set or the total vertex set. However, these bounds are far from tight for most interesting distributions.

To obtain quantitative bounds the authors turn to hypercontractivity on product spaces. For a distribution (D) on (X\times Y) they define a linear operator (T_D) mapping functions on (Y) to functions on (X) by conditional expectation. The key parameter (\delta(D)) measures how much the (L^p) norms of (T_D f) contract around (p=2):
\