A Generalization Bound for a Family of Implicit Networks

Implicit networks are a class of neural networks whose outputs are defined by the fixed point of a parameterized operator. They have enjoyed success in many applications including natural language processing, image processing, and numerous other applications. While they have found abundant empirical success, theoretical work on its generalization is still under-explored. In this work, we consider a large family of implicit networks defined parameterized contractive fixed point operators. We show a generalization bound for this class based on a covering number argument for the Rademacher complexity of these architectures.

💡 Research Summary

This paper addresses a notable gap in the theoretical understanding of implicit neural networks—models whose predictions are defined as the fixed point of a parameterized operator. While implicit architectures such as Deep Equilibrium Networks, differentiable optimization layers, and physics‑informed networks have demonstrated strong empirical performance across natural language processing, computer vision, control, and combinatorial optimization, rigorous generalization guarantees have been scarce.

The authors consider a broad family of implicit networks characterized by a contractive fixed‑point operator (T_{\psi}(\cdot;d)). For an input (d\in\mathbb{R}^m) and parameters (\theta=(\phi,\psi)\in\mathbb{R}^p), the network first solves the equation
\