Graphical model for factorization and completion of relatively high rank tensors by sparse sampling

We consider tensor factorizations based on sparse measurements of the components of relatively high rank tensors. The measurements are designed in a way that the underlying graph of interactions is a random graph. The setup will be useful in cases where a substantial amount of data is missing, as in completion of relatively high rank matrices for recommendation systems heavily used in social network services. In order to obtain theoretical insights on the setup, we consider statistical inference of the tensor factorization in a high dimensional limit, which we call as dense limit, where the graphs are large and dense but not fully connected. We build message-passing algorithms and test them in a Bayes optimal teacher-student setting in some specific cases. We also develop a replica theory to examine the performance of statistical inference in the dense limit based on a cumulant expansion. The latter approach allows one to avoid blind usage of Gaussian ansatz which fails in some fully connected systems.

💡 Research Summary

The paper addresses the problem of factorizing and completing relatively high‑rank tensors (including high‑rank matrices) when only a sparse set of measurements is available. The authors introduce a “dense limit” regime defined by N≫M≫1, where each of the N vector variables of dimension M is observed only O(M) times (c=αM with α=O(1)). This regime lies between the usual sparse‑graph setting (c=O(1)) and the fully connected case (c=O(NM)), and it captures realistic scenarios such as recommendation systems where the amount of user‑item feedback is vanishingly small compared to the total possible entries.

The measurement process is modeled on a random hyper‑graph: each hyper‑edge (a p‑plet) connects p distinct vector variables and yields a scalar observation π𝔢 = (λ/√M) Σ_{μ=1}^M F_{𝔢,μ} ∏{i∈𝔢} x{iμ}. The observation y𝔢 is generated from a likelihood P_out(y|π) which the authors instantiate as either additive Gaussian noise (y=π+w) or a sign function (y=sgn π). Two priors for the latent vectors are considered: an Ising prior (binary ±1) and a standard Gaussian prior. The linear coefficients F can be deterministic (all ones, corresponding to CP decomposition) or i.i.d. random with zero mean and unit variance; the authors show that both choices lead to identical macroscopic results in the dense limit.

The theoretical analysis proceeds via the replica method from statistical physics. By replicating the posterior distribution and performing a cumulant expansion of the interaction part of the free energy, the authors avoid the conventional Gaussian ansatz, which is known to fail for fully connected matrix factorization (p=2, M=N). The replica calculation yields explicit expressions for the free energy, order parameters (replica overlaps), and state‑equations that relate the mean‑square error (MSE) to the system parameters λ (signal strength) and γ=αp (measurement density). The analysis reveals phase transitions: for γ above a critical value γ_c the Bayes‑optimal MMSE drops sharply (information‑theoretic recovery), while for γ<γ_c the system remains in a paramagnetic (uninformative) phase. The location of γ_c depends on the prior, the noise level, and the output function; the authors provide detailed formulas for all four combinations (Ising/Gaussian × additive noise/sign output). Higher‑order transitions (first‑order, coexistence) appear especially for p≥3.

On the algorithmic side, the factor graph representation naturally leads to belief propagation (BP). The authors derive a reduced BP (r‑BP) formulation that separates messages into two stages, and then further simplify it to a generalized approximate message passing (G‑AMP) algorithm. The resulting G‑AMP updates have computational complexity O(NM) per iteration and are accompanied by a state‑evolution (SE) analysis. Crucially, the SE equations derived from G‑AMP coincide exactly with the replica state‑equations, establishing that G‑AMP attains Bayes‑optimal performance in the dense limit.

Extensive numerical experiments validate the theory. Simulations on synthetic data with N up to 10⁵ and M≈100–500 confirm that the empirical MSE follows the SE/replica predictions for both p=2 (matrix) and p=3 (3‑tensor) cases, across a range of λ and γ values. The authors compare deterministic and random F; the random spreading factors significantly improve convergence speed and stability, especially for p=2, and allow the algorithm to match theoretical predictions even for moderate system sizes. Mixed models (simultaneous p=2 and p=3 interactions) are also tested, showing that the framework extends to more complex tensor structures.

In conclusion, the paper makes three major contributions: (1) it introduces the dense‑limit scaling that enables high‑rank tensor completion from vanishingly sparse observations; (2) it provides a rigorous replica‑based analysis that overcomes the limitations of Gaussian approximations via a cumulant expansion; and (3) it presents a G‑AMP algorithm whose performance is provably optimal and empirically confirmed. These results have direct implications for large‑scale recommendation systems, social network analytics, and any application where high‑dimensional data are only partially observed. Future work may explore non‑i.i.d. priors, adaptive graph constructions, and real‑world datasets, as well as hardware‑efficient implementations of the proposed message‑passing scheme.