Provably Data-driven Multiple Hyper-parameter Tuning with Structured Loss Function

Data-driven algorithm design automates hyperparameter tuning, but its statistical foundations remain limited because model performance can depend on hyperparameters in implicit and highly non-smooth ways. Existing guarantees focus on the simple case of a one-dimensional (scalar) hyperparameter. This leaves the practically important, multi-dimensional hyperparameter tuning setting unresolved. We address this open question by establishing the first general framework for establishing generalization guarantees for tuning multi-dimensional hyperparameters in data-driven settings. Our approach strengthens the generalization guarantee framework for semi-algebraic function classes by exploiting tools from real algebraic geometry, yielding sharper, more broadly applicable guarantees. We then extend the analysis to hyperparameter tuning using the validation loss under minimal assumptions, and derive improved bounds when additional structure is available. Finally, we demonstrate the scope of the framework with new learnability results, including data-driven weighted group lasso and weighted fused lasso.

💡 Research Summary

This paper tackles a fundamental gap in the theory of data‑driven hyperparameter tuning: while existing generalization guarantees are limited to a single scalar hyperparameter, modern machine learning pipelines often involve multiple, interacting hyperparameters (e.g., elastic‑net regularization, multi‑task weighting). The authors develop a comprehensive statistical learning framework that yields provable PAC‑style guarantees for tuning an arbitrary p‑dimensional hyperparameter vector α∈ℝ^p in a bi‑level optimization setting.

The central technical contribution is a novel connection between the pseudo‑dimension of the induced loss class L={ℓ_α : X→