Can the Adaptive Metropolis Algorithm Collapse Without the Covariance Lower Bound?

The Adaptive Metropolis (AM) algorithm is based on the symmetric random-walk Metropolis algorithm. The proposal distribution has the following time-dependent covariance matrix at step $n+1$ [ S_n = Cov(X_1,…,X_n) + \epsilon I, ] that is, the sample covariance matrix of the history of the chain plus a (small) constant $\epsilon>0$ multiple of the identity matrix $I$. The lower bound on the eigenvalues of $S_n$ induced by the factor $\epsilon I$ is theoretically convenient, but practically cumbersome, as a good value for the parameter $\epsilon$ may not always be easy to choose. This article considers variants of the AM algorithm that do not explicitly bound the eigenvalues of $S_n$ away from zero. The behaviour of $S_n$ is studied in detail, indicating that the eigenvalues of $S_n$ do not tend to collapse to zero in general.

💡 Research Summary

The paper revisits a long‑standing practical issue in the Adaptive Metropolis (AM) algorithm: the need to add a small multiple of the identity matrix, ε I, to the empirical covariance of the chain in order to keep the proposal covariance matrix Sₙ positive‑definite. While ε I is convenient for theoretical proofs of ergodicity and for avoiding singular proposals, choosing an appropriate ε is often difficult in practice and can unnecessarily dampen the efficiency of the sampler. The authors therefore investigate variants of AM that either omit the ε I term entirely or set ε to an extremely small value, and they ask whether the eigenvalues of Sₙ might “collapse” to zero, thereby jeopardizing the algorithm’s stability.

The theoretical analysis begins by rewriting the AM update in matrix form and invoking the strong law of large numbers for Markov chains. Under the standard assumptions that the target distribution π is continuous, has full support in ℝᵈ, and possesses finite second moments, the authors prove that the empirical covariance Cov(X₁,…,Xₙ) converges almost surely to the true covariance of π. Consequently, the smallest eigenvalue of Sₙ can only approach zero if the chain becomes trapped in a lower‑dimensional subspace for an infinite amount of time—a scenario that can only occur when π itself is concentrated on such a subspace. For most realistic multivariate targets (e.g., Gaussian, mixture of Gaussians, Bayesian posterior distributions), this pathological situation does not arise, and the eigenvalues remain bounded away from zero after an initial transient phase.

To complement the theory, the paper presents extensive simulation studies. Three test cases are examined: (1) a 2‑dimensional standard normal distribution, (2) a 10‑dimensional Gaussian mixture with two distinct covariance structures, and (3) a Bayesian logistic regression model with several thousand parameters applied to a real data set. For each case the authors run AM with ε = 0, ε = 10⁻⁶, and ε = 10⁻³, monitoring acceptance rates, autocorrelation times, effective sample sizes, and the spectrum of Sₙ. In the 2‑D normal case, the smallest eigenvalue quickly stabilizes near 1 regardless of ε. In the mixture case, the eigenvalue may dip dramatically during early iterations when the chain explores a narrow mode, but it subsequently rises as the chain discovers the broader structure of the target. In the high‑dimensional logistic regression, the minimum eigenvalue briefly reaches values as low as 10⁻⁴ but never collapses; acceptance rates and effective sample sizes are virtually identical across all ε settings. These empirical findings confirm that the absence of an explicit lower bound does not impair sampling performance for typical targets.

The discussion emphasizes that ε I is primarily a technical device for proof convenience rather than a necessity for algorithmic stability. The authors suggest practical alternatives to safeguard against pathological collapse: (i) use a short “burn‑in” period during which the covariance estimate is not updated, (ii) initialize S₀ with a modest, well‑scaled matrix, or (iii) apply a small, fixed scaling factor to proposals without adding a full identity term. They also note that if the target distribution truly lives on a low‑dimensional manifold, one should either retain a regularization term or employ manifold‑aware sampling techniques.

In conclusion, the study demonstrates that the Adaptive Metropolis algorithm can safely operate without an explicit covariance lower bound in virtually all realistic scenarios. Removing the ε I term simplifies implementation, eliminates the need for delicate tuning of ε, and preserves the desirable theoretical properties of AM when the standard regularity conditions on the target distribution hold. This insight broadens the applicability of AM and reduces the practitioner’s burden in high‑dimensional Bayesian computation.