An Inexact Modified Quasi-Newton Method for Nonsmooth Regularized Optimization

We introduce iR2N, a modified proximal quasi-Newton method for minimizing the sum of a smooth function $f$ and a lower semi-continuous prox-bounded function $h$, allowing inexact evaluations of $f$, its gradient, and the associated proximal operators. Both $f$ and $h$ may be nonconvex. iR2N is particularly suited to settings where proximal operators are computed via iterative procedures that can be stopped early, or where the accuracy of $f$ and $\nabla f$ can be controlled, leading to significant computational savings. At each iteration, the method approximately minimizes the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term ensuring global convergence. Under standard accuracy assumptions, we prove global convergence in the sense that a first-order stationarity measure converges to zero, with worst-case evaluation complexity $O(ε^{-2})$. Numerical experiments with $\ell_p$ norms, $\ell_p$ total variation, and the indicator of the nonconvex pseudo $p$-norm ball illustrate the effectiveness and flexibility of the approach, and show how controlled inexactness can substantially reduce computational effort.

💡 Research Summary

The research paper introduces “iR2N,” a novel Inexact Modified Quasi-Newton method designed for the challenging task of minimizing the sum of a smooth function $f$ and a lower semi-continuous, prox-bounded function $h$. The core innovation lies in its ability to handle “inexactness” in the evaluation of $f$, its gradient $\nabla f$, and the associated proximal operators of $h$. This is particularly significant because, in large-scale optimization, computing exact gradients or proximal operators is often computationally prohibitive.

The proposed iR2N framework is robust enough to handle scenarios where both $f$ and $h$ are non-convex. The algorithm operates by minimizing a composite model at each iteration, consisting of a quadratic approximation of $f$, a model for $h$, and an adaptive quadratic regularization term. This adaptive regularization is a critical component that ensures global convergence by stabilizing the optimization process even in the presence of non-convexity. The authors provide a rigorous theoretical guarantee, proving that the first-order stationarity measure converges to zero with a worst-case evaluation complexity of $O(\epsilon^{-2})$.

One of the most practical advantages of iR2N is its flexibility regarding computational precision. Since the method allows for inexact evaluations, practitioners can implement “early stopping” for the iterative procedures used to compute proximal operators. This allows for a strategic trade-off: reducing computational effort in early stages or in less critical regions of the optimization landscape, and increasing precision only when necessary.

The effectiveness of the iR1N method is demonstrated through extensive numerical experiments involving $\ell_p$ norms, $\ell_p$ total variation, and the indicator of the non-convex pseudo $p$-norm ball. The results clearly illustrate that controlled inexactness can substantially reduce the total computational workload without sacrificing the ultimate convergence to a stationary point. This makes iR2N a highly promising candidate for high-dimensional, non-smooth, and non-convex optimization problems prevalent in modern machine learning, signal processing, and computer vision.