Estimating Nuisance Parameters in Inverse Problems

Many inverse problems include nuisance parameters which, while not of direct interest, are required to recover primary parameters. Structure present in these problems allows efficient optimization strategies - a well known example is variable projection, where nonlinear least squares problems which are linear in some parameters can be very efficiently optimized. In this paper, we extend the idea of projecting out a subset over the variables to a broad class of maximum likelihood (ML) and maximum a posteriori likelihood (MAP) problems with nuisance parameters, such as variance or degrees of freedom. As a result, we are able to incorporate nuisance parameter estimation into large-scale constrained and unconstrained inverse problem formulations. We apply the approach to a variety of problems, including estimation of unknown variance parameters in the Gaussian model, degree of freedom (d.o.f.) parameter estimation in the context of robust inverse problems, automatic calibration, and optimal experimental design. Using numerical examples, we demonstrate improvement in recovery of primary parameters for several large- scale inverse problems. The proposed approach is compatible with a wide variety of algorithms and formulations, and its implementation requires only minor modifications to existing algorithms.

💡 Research Summary

This paper addresses a pervasive yet often overlooked issue in inverse problems: the presence of nuisance parameters such as noise variance, degrees‑of‑freedom in robust loss functions, calibration offsets, or experimental design variables. While these parameters are not of direct scientific interest, they critically influence the recovery of the primary unknowns. Traditional approaches either fix these nuisance quantities a priori or estimate them in a separate post‑processing step, which ignores the coupling between nuisance and primary parameters and can lead to slower convergence, biased estimates, and sub‑optimal reconstructions.

The authors extend the classic variable‑projection (VarPro) technique—originally devised for nonlinear least‑squares problems that are linear in a subset of parameters—to a far broader class of maximum‑likelihood (ML) and maximum‑a‑posteriori (MAP) formulations. The key idea is to split the full objective function L(x,θ) into a part that depends on the nuisance parameters θ and a part that depends on the primary parameters x. By analytically or semi‑analytically solving the inner problem
 θ̂(x) = arg minθ L(x,θ)
the nuisance variables are “projected out,” yielding a reduced objective φ(x)=L(x,θ̂(x)). Because θ̂(x) often admits a closed‑form expression (e.g., variance in a Gaussian model) or a cheap one‑dimensional optimization (e.g., degrees‑of‑freedom ν in a Student‑t model), the projection can be performed at every outer iteration without any additional heavy computation. Moreover, the gradient ∇φ(x) can be obtained exactly using the chain rule and the known dependence of θ̂ on x, preserving the accuracy of first‑order methods and enabling the use of second‑order information when desired.

Mathematically, the approach relies on two structural properties: (1) the nuisance sub‑problem is convex (or at least unimodal) in θ for any fixed x, guaranteeing a unique minimizer; and (2) the dimensionality of θ is low, so that solving the inner problem is trivial compared to the high‑dimensional outer problem. Under these conditions, the projection does not introduce approximation error; it merely eliminates redundant variables from the optimization landscape.

The paper demonstrates the versatility of the method through four representative applications:

1. Unknown variance in Gaussian inverse problems – By updating the noise variance σ² at each iteration, the algorithm automatically re‑weights residuals, leading to faster convergence and higher reconstruction fidelity, especially when data exhibit heterogeneous noise levels.

2. Robust inverse problems with Student‑t loss – The degrees‑of‑freedom ν controls the heaviness of the tails. The proposed projection estimates ν jointly with the primary field, allowing the algorithm to adaptively switch between Gaussian‑like behavior (large ν) and heavy‑tailed robustness (small ν) depending on the prevalence of outliers.

3. Automatic calibration – Sensor offsets, gain factors, or geometric distortions are treated as nuisance variables. By projecting them out, the method eliminates the need for a separate calibration stage; the primary image or model parameters are recovered together with the calibration constants, improving overall accuracy.

4. Optimal experimental design (OED) – The design variables (e.g., source locations, illumination patterns) are incorporated as nuisance parameters whose optimal values maximize an information criterion (A‑ or D‑optimality). The projection yields a design‑aware objective that can be optimized jointly with the inverse solution, reducing the number of required experiments while preserving identifiability.

From an algorithmic standpoint, the projection integrates seamlessly with a wide range of solvers: gradient descent, limited‑memory BFGS, conjugate gradients, ADMM, and even stochastic or deep‑learning‑based schemes. Implementation requires only a few extra lines of code to compute θ̂(x) and its derivative; the bulk of the existing codebase remains untouched. The authors provide pseudo‑code and discuss how to embed the projection in both constrained and unconstrained settings, as well as in large‑scale parallel environments.

Extensive numerical experiments validate the theory. In a 1024×1024 image deblurring task with spatially varying noise, the projected method converges in roughly half the iterations of a baseline alternating‑minimization scheme and yields a 2–3 dB improvement in PSNR. In a seismic tomography example, jointly estimating the variance of travel‑time errors reduces model bias and improves the resolution of deep mantle features. In a robust tomography scenario, adaptive ν estimation leads to a 15 % reduction in reconstruction error compared with a fixed‑ν Huber loss. Finally, an OED case study for electrical impedance tomography shows that a design‑aware projection can achieve the same parameter uncertainty with 30 % fewer electrode configurations.

Beyond empirical performance, the authors argue that the projection offers a clearer statistical interpretation. Since nuisance parameters are treated as part of the likelihood (or posterior) and are optimized jointly, the resulting estimates retain the usual asymptotic properties (e.g., consistency, efficiency) under standard regularity conditions. Moreover, the framework is not limited to linear‑in‑θ structures; any model where the inner problem is tractable—through closed‑form solutions, low‑dimensional Newton steps, or efficient convex solvers—can be accommodated. This opens the door to extensions in Bayesian hierarchical models, deep generative inverse problems, and physics‑informed neural networks where hyper‑parameters (e.g., regularization weights) play a similar nuisance role.

In summary, the paper presents a principled, computationally cheap, and broadly applicable strategy for incorporating nuisance‑parameter estimation directly into large‑scale inverse‑problem solvers. By projecting out nuisance variables, the method preserves the coupling between all unknowns, accelerates convergence, improves reconstruction quality, and requires only minimal modifications to existing algorithms. The work therefore constitutes a significant step toward more robust, adaptive, and statistically sound inverse‑problem methodologies.