Learning to Discover Iterative Spectral Algorithms

We introduce AutoSpec, a neural network framework for discovering iterative spectral algorithms for large-scale numerical linear algebra and numerical optimization. Our self-supervised models adapt to input operators using coarse spectral information (e.g., eigenvalue estimates and residual norms), and they predict recurrence coefficients for computing or applying a matrix polynomial tailored to a downstream task. The effectiveness of AutoSpec relies on three ingredients: an architecture whose inference pass implements short, executable numerical linear algebra recurrences; efficient training on small synthetic problems with transfer to large-scale real-world operators; and task-defined objectives that enforce the desired approximation or preconditioning behavior across the range of spectral profiles represented in the training set. We apply AutoSpec to discovering algorithms for representative numerical linear algebra tasks: accelerating matrix-function approximation; accelerating sparse linear solvers; and spectral filtering/preconditioning for eigenvalue computations. On real-world matrices, the learned procedures deliver orders-of-magnitude improvements in accuracy and/or reductions in iteration count, relative to basic baselines. We also find clear connections to classical theory: the induced polynomials often exhibit near-equiripple, near-minimax behavior characteristic of Chebyshev polynomials.

💡 Research Summary

The paper introduces AutoSpec, a neural‑network‑driven framework for automatically discovering iterative spectral algorithms that are applicable to large‑scale numerical linear algebra (NLA) and optimization problems. Traditional algorithm‑search approaches in machine learning focus on discrete program spaces or symbolic regression, which do not translate well to NLA where algorithms are continuous, matrix‑free, and must adapt to the spectrum of the operator. AutoSpec bridges this gap by (1) extracting a cheap “spectral probe” from a new operator—typically a handful of approximate eigenvalues and associated residual norms obtained from a short Lanczos or Arnoldi run; (2) feeding this probe into a learned neural engine that predicts the coefficients of a short recurrence; and (3) executing the recurrence to apply a matrix polynomial (P(X)) to a vector. The recurrence is deliberately designed to be executable with only a few matrix‑vector products, making the whole pipeline matrix‑free and suitable for sparse, large‑scale settings.

The neural engine outputs scalar sequences ({\rho_k,\gamma_k,\eta_k,\alpha_k,\beta_k}) that define linear transition blocks (M_k). The state update (V_{k+1}=M_k V_k) yields, after (d) steps, the final action (V_{d+1}=P(X)z). Two important specializations are highlighted: (i) an affine three‑term recurrence (setting (\rho_k=0)) that subsumes classical Chebyshev, Richardson, and other polynomial acceleration schemes; and (ii) a basis‑generation plus learned expansion (setting (\gamma_k=0) for intermediate steps) where a homogeneous three‑term recurrence builds a basis of polynomials and the accumulator combines them with learned weights. Both cases map directly onto well‑studied NLA templates, showing that AutoSpec’s hypothesis space is expressive enough to capture existing methods while remaining differentiable.

Training is self‑supervised: the loss is defined solely by the downstream NLA task (e.g., residual norm for a linear solve, approximation error for a matrix function, or eigenvalue filtering error). Consequently, the model learns to trade off accuracy and iteration count directly according to the task objective. Training is performed on a synthetic dataset of small matrices with diverse spectral profiles; the learned model generalizes to real‑world large sparse matrices without further fine‑tuning, demonstrating robust spectral‑profile transfer.

Empirical evaluation covers three representative tasks: (1) matrix‑function approximation (e.g., exponential, square‑root), (2) acceleration of sparse linear solvers, and (3) spectral filtering/preconditioning for eigenvalue problems. Across a suite of real matrices from SuiteSparse, SNAP, and other repositories, AutoSpec‑derived algorithms achieve orders‑of‑magnitude reductions in iteration counts or improvements in approximation error compared with baseline Chebyshev or Richardson schemes. Notably, when the spectrum is wide or highly non‑uniform, the learned polynomials display near‑equiripple, near‑minimax behavior—essentially rediscovering the optimal Chebyshev polynomial without being explicitly programmed to do so.

The paper’s contributions are threefold: (i) a unified, differentiable representation of iterative spectral algorithms that can be learned end‑to‑end; (ii) a demonstration that coarse spectral probes are sufficient for task‑adaptive algorithm design, enabling deployment in matrix‑free environments; and (iii) empirical evidence that the learned algorithms not only match but often surpass classical theory‑driven designs, while providing insight into the underlying polynomial structure.

Future directions suggested include extending the recurrence to rational filters, handling complex spectra, integrating meta‑learning for probe design, and scaling the framework to distributed hardware where communication‑optimal recurrences are critical. AutoSpec thus opens a new pathway for automated, data‑driven algorithm discovery in the core computational kernels of scientific computing.