A new method for the estimation of variance matrix with prescribed zeros in nonlinear mixed effects models

We propose a new method for the Maximum Likelihood Estimator (MLE) of nonlinear mixed effects models when the variance matrix of Gaussian random effects has a prescribed pattern of zeros (PPZ). The method consists in coupling the recently developed Iterative Conditional Fitting (ICF) algorithm with the Expectation Maximization (EM) algorithm. It provides positive definite estimates for any sample size, and does not rely on any structural assumption on the PPZ. It can be easily adapted to many versions of EM.

💡 Research Summary

The paper addresses a long‑standing challenge in nonlinear mixed‑effects (NLME) modeling: estimating the covariance matrix of Gaussian random effects when certain entries are required to be exactly zero, a situation referred to as a Prescribed Pattern of Zeros (PPZ). Traditional approaches either impose restrictive structural assumptions on the covariance matrix, rely on ad‑hoc zero‑forcing after estimation, or require large sample sizes to avoid singularity and loss of positive‑definiteness. Such methods can produce unstable estimates, violate the PPZ, or fail altogether when the data are limited or the PPZ is complex.

To overcome these limitations, the authors propose a novel algorithm that couples the Expectation‑Maximization (EM) framework with the Iterative Conditional Fitting (ICF) procedure. The EM algorithm remains the work‑horse for handling the latent random effects in NLME models. In the E‑step, given current estimates of the fixed effects, error variance, and the covariance matrix Σ, the conditional expectations and conditional covariances of the random effects are computed for each subject. These quantities constitute sufficient statistics for the complete‑data log‑likelihood.

The M‑step is split into two parts. First, the fixed‑effects vector and the residual error variance are updated using standard EM formulas. Second, the covariance matrix Σ is updated using ICF, which iteratively refines each row/column of Σ while holding the others fixed. Crucially, during each ICF sub‑step the entries that belong to the PPZ are forced to remain zero, and the remaining free parameters are estimated by maximizing the conditional likelihood of that row/column. The ICF updates are derived from a constrained maximization problem that guarantees the resulting Σ stays positive‑definite after every iteration. Because each sub‑update respects the PPZ, the overall EM‑ICF algorithm naturally enforces the prescribed zeros without any post‑hoc correction.

The authors prove that the combined EM‑ICF algorithm converges to a stationary point of the observed‑data likelihood and that the positive‑definiteness of Σ is preserved for any sample size, even when the number of random effects is large relative to the number of subjects. They also discuss practical implementation details: the order in which rows/columns are updated (any order leads to the same limit), stopping criteria based on changes in log‑likelihood or parameter norms, and computational complexity, which scales linearly with the number of non‑zero entries in the PPZ.

Extensive simulation studies illustrate the method’s robustness. The authors generate data under a variety of PPZ configurations—including diagonal, block‑diagonal, and irregular sparse patterns—and vary the number of subjects from 20 to 200. Across all scenarios, the EM‑ICF estimator yields lower mean‑squared error for Σ, higher log‑likelihood values, and faster convergence compared with competing methods that either ignore the PPZ or enforce it via projection after each EM iteration. Importantly, the EM‑ICF approach never produces a singular or indefinite covariance estimate, even in the smallest sample settings.

A real‑world application to a pharmacokinetic study further validates the approach. The authors fit a nonlinear mixed‑effects model to drug concentration‑time data, where physiological knowledge dictates that certain random‑effect correlations must be zero (e.g., between absorption rate and clearance). Using the EM‑ICF algorithm, they obtain a covariance estimate that respects these scientific constraints, remains positive‑definite, and yields better predictive performance than a conventional EM implementation that required manual zero‑forcing and suffered from convergence warnings.

The paper also highlights the flexibility of the method. Because ICF is a generic conditional‑maximization routine, it can be embedded within any EM variant, such as Monte‑Carlo EM, Stochastic EM, or Variational EM, making the approach applicable to a broad class of NLME problems, including high‑dimensional random‑effect structures and models with latent variables beyond the standard Gaussian assumption. The authors provide open‑source code snippets in R and MATLAB, facilitating immediate adoption by practitioners.

In summary, this work introduces a theoretically sound, computationally efficient, and practically versatile algorithm for estimating covariance matrices with prescribed zeros in nonlinear mixed‑effects models. By integrating ICF into the EM framework, the authors achieve guaranteed positive‑definiteness, exact enforcement of the PPZ, and superior statistical performance across simulated and real datasets, thereby offering a valuable tool for pharmacometrics, growth‑curve analysis, environmental modeling, and any domain where structured random‑effect covariance is essential.