A Unified Framework for Lifted Training and Inversion Approaches

The training of deep neural networks predominantly relies on a combination of gradient-based optimisation and back-propagation for the computation of the gradient. While incredibly successful, this approach faces challenges such as vanishing or exploding gradients, difficulties with non-smooth activations, and an inherently sequential structure that limits parallelisation. Lifted training methods offer an alternative by reformulating the nested optimisation problem into a higher-dimensional, constrained optimisation problem where the constraints are no longer enforced directly but penalised with penalty terms. This chapter introduces a unified framework that encapsulates various lifted training strategies, including the Method of Auxiliary Coordinates, Fenchel Lifted Networks, and Lifted Bregman Training, and demonstrates how diverse architectures, such as Multi-Layer Perceptrons, Residual Neural Networks, and Proximal Neural Networks fit within this structure. By leveraging tools from convex optimisation, particularly Bregman distances, the framework facilitates distributed optimisation, accommodates non-differentiable proximal activations, and can improve the conditioning of the training landscape. We discuss the implementation of these methods using block-coordinate descent strategies, including deterministic implementations enhanced by accelerated and adaptive optimisation techniques, as well as implicit stochastic gradient methods. Furthermore, we explore the application of this framework to inverse problems, detailing methodologies for both the training of specialised networks (e.g., unrolled architectures) and the stable inversion of pre-trained networks. Numerical results on standard imaging tasks validate the effectiveness and stability of the lifted Bregman approach compared to conventional training, particularly for architectures employing proximal activations.

💡 Research Summary

This chapter presents a comprehensive treatment of lifted training—a paradigm that reformulates deep neural network learning as a higher‑dimensional constrained optimization problem in which the original layer‑wise constraints are relaxed into penalty terms. The authors first motivate the need for alternatives to the standard back‑propagation and stochastic gradient descent pipeline, highlighting issues such as vanishing/exploding gradients, the inability to handle non‑smooth activations, and the inherently sequential nature of back‑propagation that hampers parallel execution.

The core idea of lifted training is to introduce auxiliary variables for each layer’s output, denoted (u^{\ell}), and replace the exact equality (u^{\ell}= \sigma(K^{\ell}u^{\ell-1}+b^{\ell})) with a penalty such as (|u^{\ell}-\sigma(K^{\ell}u^{\ell-1}+b^{\ell})|^{2}). By doing so, gradients with respect to the network parameters no longer require differentiation of the activation function, allowing the direct use of non‑smooth proximal maps (e.g., soft‑thresholding) as activation functions.

The authors unify several previously proposed lifted methods—Method of Auxiliary Coordinates with Quadratic Penalty (MAC‑QP), Fenchel Lifted Networks, and Lifted Bregman Training—under a single mathematical formulation:

\