A Universal Class of Sharpness-Aware Minimization Algorithms

Recently, there has been a surge in interest in developing optimization algorithms for overparameterized models as achieving generalization is believed to require algorithms with suitable biases. This interest centers on minimizing sharpness of the original loss function; the Sharpness-Aware Minimization (SAM) algorithm has proven effective. However, most literature only considers a few sharpness measures, such as the maximum eigenvalue or trace of the training loss Hessian, which may not yield meaningful insights for non-convex optimization scenarios like neural networks. Additionally, many sharpness measures are sensitive to parameter invariances in neural networks, magnifying significantly under rescaling parameters. Motivated by these challenges, we introduce a new class of sharpness measures in this paper, leading to new sharpness-aware objective functions. We prove that these measures are \textit{universally expressive}, allowing any function of the training loss Hessian matrix to be represented by appropriate hyperparameters. Furthermore, we show that the proposed objective functions explicitly bias towards minimizing their corresponding sharpness measures, and how they allow meaningful applications to models with parameter invariances (such as scale-invariances). Finally, as instances of our proposed general framework, we present \textit{Frob-SAM} and \textit{Det-SAM}, which are specifically designed to minimize the Frobenius norm and the determinant of the Hessian of the training loss, respectively. We also demonstrate the advantages of our general framework through extensive experiments.

💡 Research Summary

The paper addresses a fundamental challenge in deep learning: how to design optimization algorithms that bias training toward flat minima, thereby improving generalization in over‑parameterized models. While the Sharpness‑Aware Minimization (SAM) algorithm has become a popular tool, existing work typically focuses on a narrow set of sharpness measures—most commonly the maximum eigenvalue or the trace of the Hessian of the training loss. These measures suffer from two major drawbacks. First, they are not well‑behaved for non‑convex loss landscapes because eigenvalues can be negative, making the “sharpness” interpretation ambiguous. Second, they are highly sensitive to parameter re‑parameterizations such as scaling invariances that are ubiquitous in neural networks (e.g., scaling one layer while inversely scaling another leaves the network function unchanged). Consequently, existing sharpness metrics may not capture the true geometry that matters for generalization.

Universal Sharpness Measure.
To overcome these limitations, the authors propose a parameterized family of sharpness measures, denoted as ((\phi,\psi,\mu))-sharpness: \