We study theoretical runtime guarantees for a class of optimization problems that occur in a wide variety of inference problems. these problems are motivated by the lasso framework and have applications in machine learning and computer vision. Our work shows a close connection between these problems and core questions in algorithmic graph theory. While this connection demonstrates the difficulties of obtaining runtime guarantees, it also suggests an approach of using techniques originally developed for graph algorithms. We then show that most of these problems can be formulated as a grouped least squares problem, and give efficient algorithms for this formulation. Our algorithms rely on routines for solving quadratic minimization problems, which in turn are equivalent to solving linear systems. Finally we present some experimental results on applying our approximation algorithm to image processing problems.
The problem of recovering a discrete, clear signal from noisy data is an important problem in signal processing. One general approach to this problem is to formulate an objective based on required properties of the answer, and then return its minimizer via optimization algorithms. The power of this method was first demonstrated in image denoising, where the total variation minimization approach by Rudin, Osher and Fatemi [ROF92] had much success. More recent works on sparse recovery led to the theory of compressed sensing [Can06], which includes approaches such as the least absolute shrinkage and selection operator (LASSO) objective due to Tibshirani [Tib96]. These objective functions have proven to be immensely powerful tools, applicable to problems in signal processing, statistics, and computer vision. In the most general form, given vector y and a matrix A, one seeks to minimize:
It can be shown to be equivalent to the following by introducing a Lagrangian multiplier, λ:
Many of the algorithms used to minimize the LASSO objective in practice are first order methods [NES07,BCG11], which updates a sequence of solutions using well-defined vectors related to the gradient. These methods are guaranteed to converge well when the matrix A is “well-structured”. The formal definition of this well-structuredness is closely related to the conditions required by the guarantees given in the compressed sensing literature [Can06] for the recovery of a sparse signal. As a result, these methods perform very well on problems where theoretical guarantees for solution quality are known. This good performance, combined with the simplicity of implementation, makes these algorithms the method of choice for most problems.
However, LASSO type approaches have also been successfully applied to larger classes of problems. This has in turn led to the use of these algorithms on a much wider variety of problem instances. An important case is image denoising, where works on LASSO-type objectives predates the compressed sensing literature [ROF92]. The matrices involved here are based on the connectivity of the underlying pixel structure, which is often a √ n × √ n square mesh. Even in a unweighted setting, these matrices tend to be ill-conditioned. In addition, the emergence of non-local formulations that can connect arbitrary pairs of vertices in the graph also highlights the need to handle problems that are traditionally considered ill-conditioned. We show in Appendix A that the broadest definition of LASSO problems include well-studied problems from algorithmic graph theory: Fact 1.1 Both the s-t shortest path and s-t minimum cut problems in undirected graphs can be solved by minimizing a LASSO objective.
Although linear time algorithms for unweighted shortest path are known, finding efficient parallel algorithms for this has been a long-standing open problem. The current state of the art parallel algorithm for finding 1 + ǫ approximate solutions, due to Cohen [Coh00], is quite involved. Furthermore, as the reductions done in Lemma A.1 are readily parallelizable, an efficient algorithm for LASSO minimization would also lead to an efficient parallel shortest path algorithm. This suggests that algorithms for minimizing LASSO objectives, where each iteration involve simple, parallelizable operations, are also difficult. Finding a minimum s-t cut with nearly-linear running time is also a long standing open question in algorithm design.
In fact, there are known hard instances where many algorithms do exhibit their worst case behavior [JM93]. The difficulty of these problems and the non-linear nature of the objective are two of the main challenges in obtaining fast run time guarantees for grouped least squares minimization.
Previous run time guarantees for minimizing LASSO objectives rely on general convex optimization routines [BV04], which take at least Ω(n 2 ) time. As the resolution of images are typically at least 256×256, this running time is prohibitive. As a result, when processing image streams or videos in real time, gradient descent or filtering based approaches are typically used due to time constraints, often at the cost of solution quality. The continuing increase in problem instance size, due to higher resolution of streaming videos, or 3D medical images with billions of voxels, makes the study of faster algorithms an increasingly important question.
While the connection between LASSO and graph problems gives us reasons to believe that the difficulty of graph problems also exists in minimizing LASSO objectives, it also suggests that techniques from algorithmic graph theory can be brought to bear. To this end, we draw upon recent developments in algorithms for maximum flow [CKM + 11] and minimum cost flow [DS08]. We show that relatively direct modifications of these algorithms allows us to solve a generalization of most LASSO objectives, which we term the grouped least squares problem. Our algorithm is similar to convex
This content is AI-processed based on open access ArXiv data.
