Orthogonal Approximate Message Passing Algorithms for Rectangular Spiked Matrix Models with Rotationally Invariant Noise

We propose an orthogonal approximate message passing (OAMP) algorithm for signal estimation in the rectangular spiked matrix model with general rotationally invariant (RI) noise. We establish a rigorous state evolution that exactly characterizes the high-dimensional dynamics of the algorithm. Building on this framework, we derive an optimal variant of OAMP that minimizes the predicted mean-squared error at each iteration. For the special case of i.i.d. Gaussian noise, the fixed point of the proposed OAMP algorithm coincides with that of the standard AMP algorithm. For general RI noise models, we conjecture that the optimal OAMP algorithm is statistically optimal within a broad class of iterative methods, and achieves Bayes-optimal performance in certain regimes.

💡 Research Summary

The paper tackles the problem of estimating rank‑one signals in a rectangular spiked matrix model when the additive noise is rotationally invariant (RI) rather than i.i.d. Gaussian. The observation matrix is Y = θ√(MN) u* vᵀ + W, where u∈ℝ^M and v*∈ℝ^N are unit‑norm signal vectors, θ≥0 is the signal‑to‑noise ratio, and W = U diag(σ) Vᵀ is a RI noise matrix built from two independent Haar‑distributed orthogonal matrices U and V and a diagonal matrix of singular values σ. The empirical spectral distribution of W Wᵀ converges to a deterministic measure μ with a compact support and a Hölder‑continuous density.

The authors introduce an Orthogonal Approximate Message Passing (OAMP) algorithm that extends the OAMP framework originally developed for symmetric spiked models to the rectangular setting. Each iteration consists of two parts: (i) matrix denoisers F_t and G_t that act only on the eigenvalues of Y Yᵀ and YᵀY, respectively, and are required to be “trace‑free” with respect to the limiting spectral measures μ and its companion ˜μ; (ii) entry‑wise denoisers f_t and g_t that use the entire history of past iterates together with side information a and b, and must satisfy a divergence‑free condition (zero average Jacobian). This design eliminates the Onsager correction term that appears in standard AMP, while still exploiting the spectral structure of the noise.

A central technical contribution is the derivation of three shrinkage functions ϕ₁, ϕ₂, ϕ₃ that encode how the noise spectrum should be modified at each iteration. These functions are expressed in terms of the Stieltjes transform S_μ(z) and the Hilbert transform H_μ(z) of the limiting noise spectrum, together with the signal strength θ. Lemma 1 characterizes the weak limits of several weighted empirical spectral measures (ν₁, ν₂, ν₃) that appear when the matrix denoisers are applied, and provides explicit formulas for their densities and possible point masses at the roots of 1 − θ² C(λ)=0, where C(z) is a rational function of S_μ(z).

State evolution (SE) is then established rigorously. The authors prove (Theorem 1) that for any fixed number of iterations, the joint empirical distribution of the OAMP iterates (u_t, v_t) and the true signals (u*, v*) converges in Wasserstein‑2 distance to a pair of scalar Gaussian channels: U_t = μ_{u,t} U* + σ_{u,t} Z_u, V_t = μ_{v,t} V* + σ_{v,t} Z_v, where Z_u and Z_v are standard normal and the parameters μ_{·,t}, σ_{·,t} are updated recursively using the inner products of the matrix denoisers with the spectral measures ν₁, ν₂, ν₃ and the alignment coefficients α_t = E