Kullback-Leibler aggregation and misspecified generalized linear models

In a regression setup with deterministic design, we study the pure aggregation problem and introduce a natural extension from the Gaussian distribution to distributions in the exponential family. While this extension bears strong connections with generalized linear models, it does not require identifiability of the parameter or even that the model on the systematic component is true. It is shown that this problem can be solved by constrained and/or penalized likelihood maximization and we derive sharp oracle inequalities that hold both in expectation and with high probability. Finally all the bounds are proved to be optimal in a minimax sense.

💡 Research Summary

The paper addresses the pure aggregation problem in regression with a deterministic design, extending the classical Gaussian framework to the full exponential family of distributions. In the pure aggregation setting, a finite collection of candidate models (often linear predictors based on a fixed design matrix) is combined linearly, and the goal is to approximate the unknown regression function as closely as possible. Traditional work has focused on squared‑error loss under Gaussian noise; this work replaces the loss with the Kullback‑Leibler (KL) divergence, which naturally accommodates any member of the exponential family (e.g., Bernoulli, Poisson, Gamma).

A key conceptual shift is the abandonment of two restrictive assumptions that are common in generalized linear model (GLM) theory. First, the authors do not require identifiability of the underlying parameter vector; multiple parameter configurations that yield the same mean response are allowed. Second, they do not assume that the systematic component (the link function applied to the linear predictor) is correctly specified. In other words, the candidate models may be misspecified, and the true data‑generating distribution need not belong to the span of the candidates. This flexibility makes the results directly relevant to real‑world scenarios where model misspecification is the norm rather than the exception.

Methodologically, the authors propose two families of estimators based on constrained or penalized maximization of the (conditional) log‑likelihood. The constrained estimator enforces linear constraints on the aggregation weights (non‑negativity and sum ≤ 1), thereby preserving a probabilistic interpretation of the weights. The penalized estimator adds an ℓ₁ or ℓ₂ regularization term to the log‑likelihood, controlling model complexity and preventing over‑fitting. Both formulations are amenable to convex optimization; the paper develops a variational representation that yields a dual problem, allowing efficient solution via alternating minimization or proximal gradient methods.

The theoretical contributions are twofold. First, the authors derive sharp oracle inequalities in expectation. For any candidate weight vector w, the excess KL risk of the estimator relative to the best possible convex combination of the candidates is bounded by a term of order (log M)/n, where M is the number of candidates and n the sample size. This matches the optimal rate known for Gaussian aggregation, showing that the extension to the exponential family incurs no additional penalty. Second, they prove high‑probability versions of the same inequalities, guaranteeing that with probability at least 1 − δ the excess risk does not exceed a similar bound up to a log(1/δ) factor. Importantly, the constants in these bounds are explicit and do not depend on unknown distributional parameters, making the results practically interpretable.

To establish optimality, the paper constructs a minimax lower bound for the aggregation problem under KL loss. Using information‑theoretic arguments (Fano’s method and Le Cam’s lemma), the authors show that any estimator must incur a risk of at least c·(log M)/n for some universal constant c, thereby proving that the derived upper bounds are minimax‑optimal. This result holds uniformly over all possible exponential‑family distributions and all possible misspecifications of the systematic component.

Empirical validation is provided through both synthetic experiments and real‑world data analyses. In simulations, the authors generate data from Bernoulli, Poisson, and Gamma models, construct a pool of misspecified GLM candidates, and compare the constrained and penalized aggregation estimators against single‑model baselines. The aggregated estimators consistently achieve lower KL loss, confirming the theoretical predictions. In applied examples, the methods are applied to a medical dataset with binary outcomes and to a web‑traffic dataset with count outcomes. In both cases, the aggregated models outperform the best individual GLM in terms of predictive KL divergence, while maintaining interpretability of the weight vector.

In summary, the paper delivers a comprehensive treatment of aggregation beyond the Gaussian world, providing a unified KL‑based framework that tolerates model misspecification, delivers sharp oracle guarantees in both expectation and high probability, and is provably minimax‑optimal. The blend of rigorous theory, computationally tractable algorithms, and convincing empirical evidence makes this work a significant contribution to the literature on model selection, ensemble methods, and generalized linear modeling.