Compressive sensing: a paradigm shift in signal processing

Sparse approximation has been studied for nearly a century, and it has numerous applications. Temlyakov [111] locates the first example in a 1907 paper of Schmidt [104]. In the 1950s, statisticians launched an extensive investigation of another sparse approximation problem called subset selection in regression [87] and recently least angle regression [54,113]. Later, approximation theorists began a systematic study of m-term approximation with respect to orthonormal bases and redundant systems [38,111] and very recently in [25,26].

Over the last decade, the signal processing community spurred by the work of Coifman et al. [28,29] and Mallat et al. [84,37,36] has become interested in sparse representations for compression and analysis of audio [72], images [63] and video [90]. Sparsity criteria also arise in deconvolution [110], signal modeling [100], preconditioning [74], machine learning [70], de-noising [22], regularization [33,35] and error correction [16,19,60,58,59,61]. Most sparse approximation problems employ a linear model in which the collection of elementary signals is both linearly dependent and large. These models are often called redundant or overcomplete. Recent research suggests that overcomplete models offer a genuine increase in approximation power [95,62]. Unfortunately, they also raise a serious challenge. How do we find a good representation of the input signal among the plethora of possibilities? One method is to select a parsimonious or sparse representation. The exact rationale for invoking sparsity may range from engineering to economics to philosophy. At least three justifications are commonly given:

1. It is sometimes known a priori that the input signal can be expressed as a short linear combination of elementary signals also contaminated with noise.

2. The approximation may have an associated cost that must be controlled. For example, the computational cost of evaluating the approximation depends on the number of elementary signals that participate. In compression, the goal is to minimize the number of bits required to store the approximation.

3. Some researchers cite Occam’s Razor, “Pluralitas non est ponenda sine necessitate” (causes must not be multiplied beyond necessity).

Sparse approximation problems are computationally challenging because most reasonable sparsity measures are not convex. A formal hardness proof for one important class of problems independently appeared in [88] and [36]. A vast array of heuristic methods for producing sparse approximations have been proposed, but the literature contains few guarantees of their performance. The pertinent numerical techniques fall into at least three basic categories:

1. The convex relaxation approach replaces the nonconvex sparsity measure with a related convex function to obtain a convex programming problem. The convex program can be solved in polynomial time with standard software [8], and one expects that it will yield a good sparse approximation. More on that will be said in the sequel.

2. Greedy methods make a sequence of locally optimal choices in an effort to produce a good global solution to the approximation problem. This category includes forward selection procedures (such as matching pursuits), backward selection and others. Although these approaches sometimes succeed [31,67,69,68,115,117,118], they can also fail spectacularly [40,22]. The monographs of Miller [87] and Temlyakov [111] taste the many flavors of greedy heuristic.

3. Specialized nonlinear programming software has been developed that attempts to solve sparse approximation problems directly using, for example, interior point methods [96]. These techniques are only guaranteed to discover a locally optimal solution though.

Several problems require solutions to be obtained from underdetermined systems of linear equations, i.e., systems with fewer equations than unknowns. Some example of such problems arise in linear filtering signal processing, and inverse problems. For an underdetermined system of linear equations, if there is any solution, there are infinitely many solutions. In many applications, the “simplest” solution is most acceptable. Such a solution is inspired by the minimalist principle of Occam’s Razor. For example, if the parameters of a model are being estimated then among all models that explain the data equally well, the one with the minimum number of parameters is most desirable. The notion that sparse signals-mean

…(Full text truncated)…