A survey of scalar and vector extrapolation

Scalar extrapolation and convergence acceleration methods are central tools in numerical analysis for improving the efficiency of iterative algorithms and the summation of slowly convergent series. These methods construct transformed sequences that converge more rapidly to the same limit without altering the underlying iterative process, thereby reducing computational cost and enhancing numerical accuracy. Historically, the origins of such techniques can be traced back to classical algebraic methods by AlKhwarizmi and early series acceleration techniques by Newton, while systematic approaches emerged in the 20th century with Aitken process and Richardson extrapolation. Later developments, including the Shanks transformation and Wynn epsilon algorithm, provided general frameworks capable of eliminating multiple dominant error components, with deep connections to Pade approximants and rational approximations of generating functions. This paper presents a comprehensive review of classical scalar extrapolation methods, including Richardson extrapolation, Aitken process, Shanks transformation, Wynn epsilon algorithm, and other algorithms. We examine their theoretical foundations, asymptotic error models, convergence properties, numerical stability, and practical implementation considerations. The second part of this work is dedicated to vector extrapolation methods: polynomial based ones and epsilon algorithm generalizations to vector sequences. Additionally, we highlight modern developments such as their applications to iterative solvers, Krylov subspace methods, and large-scale computational simulations. The aim of this review is to provide a unified perspective on scalar and vector extrapolation techniques, bridging historical origins, theoretical insights, and contemporary computational applications.

💡 Research Summary

The paper provides a comprehensive review of scalar and vector extrapolation (convergence acceleration) techniques, tracing their historical development, theoretical foundations, and modern computational applications. It begins with a historical overview, noting early algebraic approximations by Al‑Khwarizmi, series acceleration ideas of Newton and Gregory, and the systematic emergence of Aitken’s Δ² process and Richardson extrapolation in the early 20th century. The mid‑century introduction of the Shanks transformation and Wynn’s ε‑algorithm is highlighted as a major step toward handling sequences with multiple exponential error components, and the deep connection of these transformations to Padé approximants is explained.

The scalar section (Section 2) examines the most widely used methods in detail. Richardson extrapolation is presented as a technique that assumes an asymptotic expansion x(h)=x*+c₁hᵖ+… and eliminates the leading hᵖ term by linear combination of approximations at different step sizes; recursive application yields high‑order schemes such as Romberg integration. Aitken’s Δ² process is analyzed under the assumption of linear convergence sₙ₊₁−s*≈λ(sₙ−s*) with |λ|<1; the method’s kernel consists of sequences of the form sₙ=s*+cλⁿ, and it converts linear convergence into quadratic convergence, albeit with potential numerical instability when the second difference becomes small. The Shanks transformation is introduced as a determinant‑based generalization capable of annihilating several exponential terms simultaneously, and Wynn’s ε‑algorithm is shown to implement the same transformation recursively, avoiding costly determinant evaluations and providing a numerically stable framework. The paper also discusses the ρ‑ and θ‑algorithms as further variants, comparing their error models and convergence conditions.

Section 3 shifts focus to vector and matrix extrapolation. Reduced Rank Extrapolation (RRE) and Minimal Polynomial Extrapolation (MPE) are described as methods that construct Krylov‑type subspaces from past iterates, with RRE minimizing the norm of a linear combination of error vectors and MPE minimizing the degree of the annihilating polynomial. The Modified Minimal Polynomial Extrapolation (MMPE) improves numerical stability of MPE by employing QR or LU factorizations. Vector ε‑Algorithm (VEA) and Topological ε‑Algorithm (TEA) extend Wynn’s scalar ε‑algorithm to multi‑dimensional sequences, using multi‑dimensional differences to cancel dominant error components. Anderson Acceleration (AA) is presented as a modern, widely used technique for nonlinear fixed‑point iterations; it stores the last m residuals and solves a small least‑squares problem to generate an accelerated update, effectively acting as a nonlinear analogue of MPE.

For each method the authors detail the underlying error model (typically eₙ≈∑cⱼvⱼλⱼⁿ), convergence theorems, numerical stability considerations (condition numbers, round‑off sensitivity), and computational complexity (e.g., O(n m) versus O(m³) for solving the small auxiliary systems). Practical implementation issues such as scaling, regularization, and stopping criteria are discussed.

Section 4 presents numerical experiments that benchmark the discussed algorithms on a variety of problems: large sparse linear systems, nonlinear root‑finding via Newton‑Krylov, and slowly convergent series such as the Riemann ζ‑function expansion. Results demonstrate that vector extrapolation (RRE, MPE, AA) can dramatically reduce iteration counts and wall‑clock time compared with unaccelerated methods, while maintaining modest memory footprints. In a high‑dimensional fluid dynamics simulation, combining Anderson acceleration with MPE achieved more than a 30 % speed‑up over a standard Newton‑Krylov solver. The experiments also illustrate the trade‑offs between robustness (e.g., ε‑algorithm’s stability) and aggressiveness (e.g., Anderson’s rapid convergence but sensitivity to parameter m).

The conclusion emphasizes the deep theoretical links between scalar and vector extrapolation—both can be viewed through the lens of Padé approximants and Krylov subspace theory—and argues that, despite the rise of machine‑learning‑based solvers, classical extrapolation remains a vital tool for improving efficiency and accuracy in large‑scale scientific computing. Future research directions suggested include data‑driven error‑model identification, adaptive selection of extrapolation parameters, and scalable parallel implementations for exascale architectures.