Operator Splitting with Hamilton-Jacobi-based Proximals

Operator splitting algorithms are a cornerstone of modern first-order optimization, decomposing complex problems into simpler subproblems solved via proximal operators. However, most functions lack closed-form proximal operators, which has long restricted these methods to a narrow set of problems. Hamilton-Jacobi-based proximal operator (HJ-Prox) is a recent derivative-free Monte Carlo technique based on Hamilton-Jacobi PDE theory, that approximates proximal operators numerically. In this work, we introduce a unified framework for operator splitting via HJ-Prox, which allows for deployment of operator splitting even when functions are not proximable. We prove that replacing exact proximal steps with HJ-Prox in algorithms such as proximal point, proximal gradient descent, Douglas-Rachford splitting, Davis-Yin splitting, and primal-dual hybrid gradient preserves convergence guarantees under mild assumptions. Numerical experiments demonstrate HJ-Prox is competitive and effective on a wide variety of statistical learning tasks.

💡 Research Summary

This paper addresses a fundamental limitation of modern operator‑splitting methods: the reliance on closed‑form proximal operators. While algorithms such as the proximal point method, proximal gradient descent, Douglas‑Rachford splitting, Davis‑Yin splitting, and primal‑dual hybrid gradient enjoy wide use, they become impractical when the constituent functions lack analytically tractable proximals. The authors propose to replace exact proximal steps with a derivative‑free Monte‑Carlo approximation called Hamilton‑Jacobi proximal (HJ‑Prox). HJ‑Prox stems from Hamilton‑Jacobi partial differential equation theory and approximates the proximal map by sampling from a Gaussian distribution centered at the current iterate, weighting samples with (\exp(-f(y)/\delta)). The smoothing parameter (\delta>0) controls bias, and as (\delta\to0) the approximation converges to the true proximal operator.

The core contribution is a unified convergence framework that treats each splitting algorithm as a perturbed fixed‑point iteration. Building on the Krasnosel’skiĭ‑Mann theorem for averaged operators, the authors show that if the sequence of approximation errors is almost surely summable, the iterates converge to a solution of the original problem. Two key error analyses are provided: (1) a deterministic bound (| \operatorname{prox}{\delta t f} - \operatorname{prox}{t f}| \le \sqrt{n t \delta}) (Crandall‑Lions, 1983), and (2) a probabilistic Monte‑Carlo bound that depends exponentially on the Lipschitz constant of the function via the factor (J^\star = \exp(2L^2 t\delta)). By selecting sequences ({\delta_k}), ({t_k}), and sample sizes ({N_k}) that satisfy mild summability conditions, the authors guarantee almost‑sure convergence for all five algorithms.

For the proximal point method and proximal gradient descent, the step size (t_k) is allowed to decay as (O(1/k)); the authors propose (\delta_k = k^{-(p+1)}), (\alpha_k = k^{-(p+2)}) (with (p>0)) and a sample size (N_k = O(e^{kp}k^{p+2})). Although the theoretical sample complexity is sub‑exponential, in practice a modest fixed sample size (e.g., (N=1000)) suffices. For Douglas‑Rachford, Davis‑Yin, and PDHG, a constant step size is required, and the analysis instead imposes summability on (\sqrt{\delta_k}) and the tail probability (\alpha_k).

The paper presents detailed convergence theorems (Theorems 3.5–3.10) for each HJ‑Prox‑based algorithm, showing almost‑sure convergence to a minimizer under the stated assumptions. A notable insight is that operator splitting dramatically reduces the Monte‑Carlo constant: applying HJ‑Prox to the composite objective (f+g) incurs a factor (\exp(2(L_f+L_g)^2 t\delta)), whereas splitting allows separate approximations with factors (\exp(2L_f^2 t\delta)) and (\exp(2L_g^2 t\delta)), yielding a much lower variance and sample requirement.

Empirical validation covers seven nonsmooth optimization tasks, including LASSO, sparse group LASSO, and image denoising. In each case, HJ‑Prox‑augmented splitting matches the objective trajectories and solution quality of exact‑proximal baselines, even when the exact proximal operator is unavailable. The experiments use a decreasing (\delta_k) schedule and a fixed Monte‑Carlo sample size, confirming that the theoretical conditions are conservative and that practical performance is robust.

In summary, the authors demonstrate that Hamilton‑Jacobi‑based proximal approximations can be seamlessly integrated into a broad class of operator‑splitting algorithms, preserving their convergence guarantees while eliminating the need for analytically tractable proximal operators. This work opens the door to applying splitting methods to a much wider range of problems, including those with black‑box or highly complex regularizers, and suggests future directions such as adaptive sampling schemes, variance‑reduced Monte‑Carlo estimators, and large‑scale distributed implementations.