Worst-case Compressibility of Discrete and Finite Distributions

In the worst-case distributed source coding (DSC) problem of [1], the smaller cardinality of the support-set describing the correlation in informant data, may neither imply that fewer informant bits are required nor that fewer informants need to be queried, to finish the data-gathering at the sink. It is important to formally address these observations for two reasons: first, to develop good worst-case information measures and second, to perform meaningful worst-case information-theoretic analysis of various distributed data-gathering problems. Towards this goal, we introduce the notions of bit-compressibility and informant-compressibility of support-sets. We consider DSC and distributed function computation problems and provide results on computing the bit- and informant- compressibilities regions of the support-sets as a function of their cardinality.

💡 Research Summary

The paper tackles the worst‑case scenario of distributed source coding (DSC), challenging the common intuition that a smaller support‑set (the set of all possible joint observations across informants) automatically leads to lower communication costs or fewer queried informants. To formalize this intuition, the authors introduce two novel worst‑case information measures: bit‑compressibility and informant‑compressibility.

Bit‑compressibility quantifies the minimum number of bits that must be transmitted from the informants to the sink in order to guarantee exact reconstruction of the joint data for every element of the support‑set. Formally, for a support‑set (S), the measure is denoted (B_{\min}(S)). While the naïve lower bound is (\log_2|S|), the authors show that structural dependencies among variables can raise this bound substantially. For example, even when (|S|) is small, if certain variables are deterministic functions of others in a non‑linear way, the sink may still need to receive nearly the full entropy of each variable, inflating (B_{\min}).

Informant‑compressibility, denoted (I_{\min}(S)), captures the smallest number of informants that must be actively queried to enable exact reconstruction of the joint data. This measure is independent of the total number of bits; it reflects the combinatorial geometry of the support‑set. The paper proves that for some support‑sets, especially those exhibiting special combinatorial patterns (e.g., Latin squares, partial permutations), (I_{\min}) can be dramatically lower than the total number of informants, even when (|S|) is large. Conversely, for “full‑cube” support‑sets where every possible combination appears, (I_{\min}) equals the total number of informants.

The authors develop tight upper and lower bounds for both (B_{\min}(|S|)) and (I_{\min}(|S|)) using tools from combinatorics, algebraic structures (groups, fields), and information theory. They provide explicit constructions of support‑sets that achieve the extremal points of these bounds, thereby illustrating that cardinality alone is insufficient to characterize worst‑case compression performance.

Two concrete application domains are examined.

1. Distributed Source Coding (DSC) – Here each informant compresses its observation and sends it to a central sink. The paper shows that minimizing (B_{\min}) leads to coding schemes that adapt to the internal dependencies of the support‑set, while minimizing (I_{\min}) yields query strategies that activate only a subset of sensors, saving energy and bandwidth.

2. Distributed Function Computation – The goal is to compute a function (e.g., sum, max, average) of the joint data rather than reconstruct the data itself. The authors extend the compressibility concepts to a “function‑specific compressibility” framework, proving that for many useful functions the required number of bits and informants can be far lower than in the full‑reconstruction case. For instance, computing a sum often only needs the partial sums from a few strategically chosen informants, dramatically reducing both (B_{\min}) and (I_{\min}).

The paper concludes by emphasizing the practical relevance of these worst‑case measures. In sensor networks with stringent energy constraints, blockchain or distributed ledger systems requiring minimal verification data, and privacy‑preserving data collection where only aggregate statistics are needed, the proposed framework offers a principled way to design communication protocols that are robust against the most adverse data realizations. By decoupling support‑set cardinality from structural dependencies, the work opens a new line of research into worst‑case information‑theoretic limits for a broad class of distributed data‑gathering problems.