Controlling overestimation of error covariance in ensemble Kalman filters with sparse observations: A variance limiting Kalman filter

We consider the problem of an ensemble Kalman filter when only partial observations are available. In particular we consider the situation where the observational space consists of variables which are directly observable with known observational error, and of variables of which only their climatic variance and mean are given. To limit the variance of the latter poorly resolved variables we derive a variance limiting Kalman filter (VLKF) in a variational setting. We analyze the variance limiting Kalman filter for a simple linear toy model and determine its range of optimal performance. We explore the variance limiting Kalman filter in an ensemble transform setting for the Lorenz-96 system, and show that incorporating the information of the variance of some un-observable variables can improve the skill and also increase the stability of the data assimilation procedure.

💡 Research Summary

The paper addresses a common challenge in data assimilation: the presence of partially observed systems where only a subset of state variables can be directly measured, while the remaining variables are unobservable but have known climatological means and variances. Traditional ensemble Kalman filters (EnKFs) handle the observable variables using standard observation operators, but they rely solely on model forecasts for the unobserved components. In sparse observation networks, this can lead to uncontrolled growth of error covariances for the unobserved variables, especially when covariance inflation is applied, potentially causing filter divergence or unrealistic analysis spreads.

To mitigate this problem, the authors propose a Variance Limiting Kalman Filter (VLKF) formulated within a variational framework. The state vector z is partitioned into observable components x (dimension n) and pseudo‑observable components y (dimension m). The observation operator H maps the full state to the observable space, while a second operator h maps the state to the pseudo‑observable space. For x, standard Gaussian observation errors with covariance Rₒ are assumed. For y, only the climatological mean a_clim and covariance A_clim are known; no direct measurements are available.

The VLKF introduces a weak‑constraint “pseudo‑observation” term into the cost function: \