Pseudo-Invertible Neural Networks

The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems. In this paper, we propose a natural generalization of PInv to the nonlinear regime in general and to neural networks in particular. We introduce Surjective Pseudo-invertible Neural Networks (SPNN), a class of architectures explicitly designed to admit a tractable non-linear PInv. The proposed non-linear PInv and its implementation in SPNN satisfy fundamental geometric properties. One such property is null-space projection or “Back-Projection”, $x’ = x + A^\dagger(y-Ax)$, which moves a sample $x$ to its closest consistent state $x’$ satisfying $Ax=y$. We formalize Non-Linear Back-Projection (NLBP), a method that guarantees the same consistency constraint for non-linear mappings $f(x)=y$ via our defined PInv. We leverage SPNNs to expand the scope of zero-shot inverse problems. Diffusion-based null-space projection has revolutionized zero-shot solving for linear inverse problems by exploiting closed-form back-projection. We extend this method to non-linear degradations. Here, “degradation” is broadly generalized to include any non-linear loss of information, spanning from optical distortions to semantic abstractions like classification. This approach enables zero-shot inversion of complex degradations and allows precise semantic control over generative outputs without retraining the diffusion prior.

💡 Research Summary

The paper “Pseudo‑Invertible Neural Networks” proposes a principled extension of the Moore‑Penrose pseudo‑inverse (PInv) from linear algebra to the nonlinear regime, with a focus on deep neural networks. The authors first observe that while the PInv in linear systems enjoys four Penrose identities—two guaranteeing reflexive consistency (AA†A = A, A†AA† = A†) and two enforcing orthogonal projection (AA† and A†A are Hermitian)—only the first two can be meaningfully generalized to arbitrary nonlinear maps because the notion of an adjoint does not exist in general. To obtain a unique nonlinear pseudo‑inverse, they introduce the concept of a Bijective Completion: a surjective map g : X→Y is lifted to a diffeomorphism G : X→Y×Z by appending auxiliary coordinates q(x). The pseudo‑inverse g† is then defined as the element of the pre‑image of y that minimizes the Euclidean distance to the origin in the completed space, i.e.
 g†(y) = arg minₓ∈g⁻¹(y) ‖G(x) − G(0)‖₂.
This definition preserves three essential properties: (1) reflexive consistency (the first two Penrose identities hold by construction), (2) coordinate consistency (if g is a composition of a diffeomorphic coordinate change and a linear surjection, the pseudo‑inverse reduces to the linear Moore‑Penrose inverse expressed in the transformed coordinates), and (3) back‑projection consistency (a closed‑form non‑linear back‑projection (NLBP) can be derived that acts as an orthogonal projection in the G‑metric).

To make this theory practical, the authors design Surjective Pseudo‑invertible Neural Networks (SPNNs). The core building block is an Affine Surjective Coupling Layer: an input vector x∈ℝᴰ is split into x₀∈ℝᵈ and x₁∈ℝᴰ⁻ᵈ. The high‑dimensional part x₁ modulates x₀ through learned scale s(x₁) and translation t(x₁) functions, producing a lower‑dimensional output y = x₀ ⊙ s(x₁) + t(x₁). Because dimensionality is reduced (d < D), information is lost, making the forward map surjective but not bijective. The missing component x₁ is recovered by a deterministic auxiliary network r : ℝᵈ→ℝᴰ⁻ᵈ that predicts x₁ from y. The inverse is then computed as
 \hat{x}_1 = r(y), \hat{x}_0 = y − t(\hat{x}_1) · s(\hat{x}_1), \hat{x} =