Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons

A lossy data compression scheme for uniformly biased Boolean messages is investigated via statistical mechanics techniques. We utilize tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are non-monotonic. The scheme performance at the infinite code length limit is analyzed using the replica method. Both committee and parity treelike networks are shown to saturate the Shannon bound. The AT stability of the Replica Symmetric solution is analyzed, and the tuning of the non-monotonic transfer function is also discussed.

💡 Research Summary

The paper investigates a lossy compression scheme for uniformly biased Boolean messages by employing tree‑structured neural networks with non‑monotonic transfer functions. Two specific architectures are studied: the committee tree, which aggregates the outputs of L hidden perceptrons by majority vote, and the parity tree, which aggregates them by an XOR (parity) operation. In both cases each hidden unit computes the sign of a weighted sum of the input bits passed through a non‑monotonic activation φ(x;α) that switches between +1 and –1 at one or more thresholds controlled by a tunable parameter α. The encoder maps an N‑bit source vector s∈{±1}^N onto an M‑bit codeword y∈{±1}^M (with compression rate R=M/N<1), while the decoder reconstructs an estimate ŝ from y. The performance metric is the average Hamming distortion D=⟨(s_i−ŝ_i)^2⟩/4, which must stay below a prescribed level.

To evaluate the asymptotic performance (N→∞) the authors apply the replica method from statistical physics. They introduce n replicas of the system, compute the disorder‑averaged replicated partition function, and take the limit n→0. Under the replica‑symmetric (RS) ansatz the order parameters reduce to a single overlap q between different replicas and its conjugate (\hat q). The resulting free‑energy functional yields self‑consistency equations that are equivalent to a set of saddle‑point conditions for q and (\hat q). Solving these equations analytically, the authors obtain the rate‑distortion relation for both architectures. Remarkably, the derived expression coincides exactly with Shannon’s bound for binary sources with bias:

\