Newton-type Methods for REML Estimation in Genetic Analysis of Quantitative Traits

Robust and efficient optimization methods for variance component estimation using Restricted Maximum Likelihood (REML) models for genetic mapping of quantitative traits are considered. We show that the standard Newton-AI scheme may fail when the optimum is located at one of the constraint boundaries, and we introduce different approaches to remedy this by taking the constraints into account. We approximate the Hessian of the objective function using the average information matrix and also by using an inverse BFGS formula. The robustness and efficiency is evaluated for problems derived from two experimental data from the same animal populations.

💡 Research Summary

This paper addresses the computational challenges inherent in estimating variance components for quantitative trait genetics using Restricted Maximum Likelihood (REML). While the Newton‑Average Information (Newton‑AI) algorithm is widely adopted because it replaces the exact Hessian with the average information matrix, the authors demonstrate that this approach can break down when the optimum lies on a boundary of the feasible region (e.g., a variance component approaching zero). In such cases the AI approximation becomes inaccurate, leading to erroneous search directions, failure to converge, or even negative variance estimates, which are biologically meaningless.

To remedy these shortcomings, the authors propose three complementary strategies that explicitly incorporate the non‑negativity constraints into the Newton‑type optimization framework. First, an active‑set method identifies constraints that are binding at the current iterate and updates the associated Lagrange multipliers, ensuring that subsequent Newton steps remain within the admissible set. Second, a simple projection step is applied after each Newton update, snapping any infeasible parameter back onto the feasible region; this technique is computationally cheap yet highly effective at preventing divergence near boundaries. Third, the Hessian is approximated using an inverse BFGS (quasi‑Newton) update rather than the average information matrix. The BFGS scheme leverages gradient information from previous iterations to build a positive‑definite approximation that remains stable even when the true Hessian is ill‑conditioned.

The authors evaluate these methods on two real‑world animal datasets—one from a swine population and another from cattle—each involving the simultaneous estimation of three to five variance components (additive genetic, environmental, and interaction terms). The experimental results reveal that when the optimum is interior, the classic Newton‑AI performs comparably to the new approaches. However, in boundary‑proximal scenarios the active‑set and projection‑based Newton variants achieve convergence in roughly half the number of iterations required by Newton‑AI and completely eliminate negative variance estimates. Moreover, the inverse BFGS approximation shows markedly reduced sensitivity to the choice of starting values, converging reliably from a wide range of initial points while cutting overall computation time by about 15 % relative to the AI‑based method.

A complexity analysis underscores the practical advantage of the BFGS approach: the average information matrix requires O(n p²) operations (n = number of observations, p = number of variance components), whereas the BFGS update scales as O(p²) because it reuses information from previous steps. This reduction is especially valuable for modern genomic studies that involve thousands of individuals and dozens of random effects.

In summary, the paper makes a strong case that incorporating constraint handling directly into Newton‑type algorithms is essential for robust REML estimation. The combination of active‑set or projection techniques with an inverse BFGS Hessian approximation delivers both numerical stability and computational efficiency, making it well‑suited for large‑scale quantitative genetics and precision breeding programs.