Incoherent dictionaries and the statistical restricted isometry property

1 Incoherent dictionarie s and the statistical restricte d isometry property Shamgar Gure vich and Ronn y Hadani Abstract — In this article we present a statistical version of the Cand ` es-T ao restricted isometry property (SRIP for short) wh ich holds in general for any in coherent dictionary which is a disjoint union of orthonormal bases. In addition, und er appropriate normalization, the eigen v alues of the associated Gram matrix fluctuate around λ = 1 according to the Wigner semicircle distribution. The result is then applied to v arious dictionaries that arise n aturally in the settin g of finite harmonic analysis, giving, in p articular , a better understandin g on a remark of Applebaum- Howard-Searle-Calderbank concerning RIP for the Heisenberg dictionary of chirp like fu nctions. Index T erms — Incoherent dictionaries, Statistical RIP , Wigner semicircle distribution, d eterministic examples, Heisenberg-W eil repre sentation. I . I N T RO D U C T I O N D igital signals, or simply signals, can be though t of as complex valued functions on the finite field F p , whe re p is a prime number . The space of sign als H = C ( F p ) is a Hilbert space of d imension p , with the inner p roduct given by the standard form ula h f , g i = P t ∈ F p f ( t ) g ( t ) . A dictio nary D is simply a set of vector s (also called atoms ) in H . The number of vectors in D can exceed the dimension of the H ilbert spac e H , in fact, th e mo st interesting situation is when | D | ≫ p = dim H . In this set- up we define a r esolution of the H ilbert spac e H via D , which is th e morph ism of vector spaces Θ : C ( D ) → H , giv en by Θ ( f ) = P ϕ ∈ D f ( ϕ ) ϕ , for every f ∈ C ( D ) . A more concrete way to think of th e morphism Θ is as a p × | D | matrix with the colum ns being the atoms in D . In th e last two de cades [11], and in particular in recen t years [3], [4], [5], [6], [7], [8], resolution s of Hilbert spac es became an importan t too l in signal processing, in particula r in the emerging the ories o f spar sity and co mpressive sensing. I I . T H E R E S T R I C T E D I S O M E T RY P RO P E RT Y A usefu l prope rty of a r esolution is the restricted isom etry proper ty ( RIP f or sho rt) d efined by Cand ` es-T ao in [7]. Fix a natural num ber n ∈ N and a pair of positive real numbers δ 1 , δ 2 ∈ R > 0 . S. Gure vi ch is with the De partment of Mathe matics, Uni versi ty of Califor - nia, Berkele y , CA 94720, US A. Em ail: shamgar@math.berk ele y .edu. R. Hadani is with the D epartme nt of Mathematic s, Univ ersity of Chica go, IL 60637, USA. Email: hadani@math.uc hicago.e du. Date: Sep. 1, 2008. Definition II-.1: A dictionary D satisfies the r estricted isometry p r operty with coefficients ( δ 1 , δ 2 , n ) if for every su bset S ⊂ D such that | S | ≤ n we have (1 − δ 2 ) k f k ≤ k Θ ( f ) k ≤ (1 + δ 1 ) k f k , for every fu nction f ∈ C ( D ) which is supporte d on the set S . Equiv alently , RIP ca n be form ulated in terms of the spe ctral radius of th e correspo nding Gra m o perator . Let G ( S ) deno te the comp osition Θ ∗ S ◦ Θ S with Θ S denoting the r estriction of Θ to the subspace C S ( D ) ⊂ C ( D ) of fun ctions sup ported on the set S . The dictiona ry D satis fies ( δ 1 , δ 2 , n ) -RIP if f or ev ery subset S ⊂ D suc h th at | S | ≤ n we ha ve δ 2 ≤ k G ( S ) − I d S k ≤ δ 1 , where I d S is th e identity oper ator on C S ( D ) . It is known [2], [8] that th e RIP hold s for rando m diction ar- ies. Howev e r , on e would like to add ress the f ollowing problem [1], [ 10], [9], [20], [21], [22], [2 3], [2 5], [2 4], [2 6], [2 7]: Pr oblem II-. 2: Find d eterministic construction of a dictio- nary D with | D | ≫ p which satisfies RIP with co efficients in the cr itical regime δ 1 , δ 2 ≪ 1 and n = α · p , (II-.1 ) for some constant 0 < α < 1 . I I I . I N C O H E R E N T D I C T I O N A R I E S Fix a p ositi ve real n umber µ ∈ R > 0 . The fo llowing notion was introd uced in [9], [12] and was used to study similar problem s in [ 26], [27]: Definition III-.3 : A dictionary D is called in coherent with coheren ce coefficient µ (also called µ -coherent) if for every pair of distinct atoms ϕ, φ ∈ D |h ϕ, φ i| ≤ µ √ p . In this article we will explore a gene ral re lation between RIP and inco herence. Our moti vation comes from three examples of inco herent dictionar ies which arise naturally in the setting of finite harmo nic analysis: • The fi rst e x ample [1 8], [1 9], re ferred to as th e Heisen - ber g diction ary D H , is constru cted using th e Heisen berg representatio n o f the finite He isenberg group H ( F p ) . The Heisenberg dictio nary is of size approx imately p 2 and its coheren ce co efficient is µ = 1 . • The second e xample [15], [16], [1 7], which is refer red to as the oscillator d ictionary D O , is co nstructed using the W eil rep resentation of the fin ite symp lectic grou p 2 S L 2 ( F p ) . The oscillator d ictionary is of size approxi- mately p 3 and its coheren ce coefficient is µ = 4 . • The th ird example [15], [1 6], [ 17], referr ed to as th e extended oscillator d ictionary D E O , is constru cted using the Heisen berg-W eil representation [28], [ 13] of the finite Jacobi grou p, i.e. , the semi-dir ect produ ct J ( F p ) = S L 2 ( F p ) ⋉ H ( F p ) . The extended oscillator d ictionary is o f size a pprox imately p 5 and its coherence coefficient is µ = 4 . The three examples of dictionaries we just describ ed con- stitute reasonable candidates fo r solving Pro blem II-.2: They are large in the sense that | D | ≫ p, and emp irical evidences suggest (see [1] for the case of D H ) that they migh t satisfy RIP with co efficients in the critical regime (II -.1). W e summarize this as follows: Pr oblem III- .4: Do the d ictionaries D H , D O and D E O sat- isfy the RIP with coefficients δ 1 , δ 2 ≪ 1 and n = α · p , for some 0 < α < 1 ? I V . M A I N R E S U LT S In this article we for mulate a relaxed statistical version of RIP , called statistical iso metry prop erty (SRIP for short) which holds for any in coherent d ictionary D whic h is, in ad dition, a disjoint union of orth onorm al bases: D = a x ∈ X B x , (IV - .2) where B x =  b 1 x , .., b p x  is an o rthono rmal basis of H , f or ev ery x ∈ X . A. The statistical r e stricted iso metry pr operty Let D be an inco herent dictionar y o f the f orm (I V -.2). Roughly , th e statem ent is that for S ⊂ D , | S | = n with n = p 1 − ε , fo r 0 < ε < 1 , chosen unifor mly at rand om, the operator n orm k G ( S ) − I d S k is small with high prob ability . Precisely , we ha ve Theor em IV -A .1 ( SRIP p r operty [14]): For every k ∈ N , there exists a co nstant C ( k ) such that the probability Pr  k G ( S ) − I d S k ≥ p − ε/ 2  ≤ C ( k ) p 1 − εk/ 2 . (IV - A.1) The above theorem , in particular, imp lies that probability Pr  k G ( S ) − I d S k ≥ p − ε/ 2  → 0 as p → ∞ faster then p − l for any l ∈ N . B. The statistics of the eigen values A na tural th ing to know is how the eigenv alu es of th e Gra m operator G ( S ) fluctuate aro und 1 . In this regard , we stud y th e spectral statistics of the nor malized err or term E ( S ) = ( p/n ) 1 / 2 ( G ( S ) − I d S ) . Let ρ E ( S ) = n − 1 P n i =1 δ λ i denote th e spectra l distribution of E ( S ) where λ i , i = 1 , .., n , are the real eigenv alu es of the Her mitian o perator E ( S ) . The f ollowing theorem ass erts that ρ E conv erges i n pro bability as p → ∞ to th e W ign er semicircle d istribution ρ S C ( x ) = (2 π ) − 1 √ 4 − x 2 · 1 [2 , − 2] ( x ) where 1 [2 , − 2] is th e char acteristic function of the interval [ − 2 , 2] . Theor em IV -B .1 (Semicircle distribution [14]): W e have lim p →∞ ρ E Pr = ρ S C . (IV - B.1) Remark IV -B.2: A limit of the form (IV -B.1) is familiar in random matrix theo ry as the asym ptotic of the spectral distri- bution of W ign er matrices. In terestingly , the same asymp totic distribution appears in o ur situation, albeit, the p robability spaces are of a different nature (our probability spaces are, in p articular, muc h smaller ). In particular, Theo rems IV -A.1, IV -B.1 can b e applied to the three examples D H , D O and D E O , which are all of the approp riate f orm (IV -.2). Finally , o ur result gives n ew infor- mation o n a r emark of A pplebaum -How ard-Searle- Calderbank [1] concer ning RIP of the Heisenb erg d ictionary . Remark IV -B.3: For practical app lications, it might be im- portant to c ompute e x plicitly th e constants C ( k ) which ap- pears in (IV -A.1). This constan t depends on the inco herence coefficient µ , therefo re, for a fixed p , having µ as small as possible is preferab le. Ackno wledgement. It is a pleasure to thank our teacher J. Bern stein for his con tinuos support. W e are gratefu l to N. Sochen fo r many im portant d iscusiions. W e than k F . Bruckstein, R. Calderba nk, M. Elad, Y . Eldar, an d A. Sahai fo r sharing with us some of their thoughts about s ignal processing. W e are gr ateful to R. Howe, A. Man, M. Revzen an d Y . Za k for explaining us th e no tion of mutually unbiased bases. R E F E R E N C E S [1] Appleba um L. , How ard S., Searle S. , and Calderba nk R., Chirp sensing codes: Deterministi c compressed s ensing measurement s for fast recov- ery . (Preprint, 2008). [2] Barani uk R., Davenpo rt M., DeV ore R.A. and W akin M.B., A simple proof of the restricte d isometry property for random matrice s. Construc- tive Approxi mation, to appear (2007). [3] Bruckste in A.M., D onoho D.L. and Elad M., ”From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images”, to appear in SIAM Revie w (2007). [4] Compressi ve Sensing Resourc es. A vai lable at http:/ /www .dsp.ece.rice.ed u/cs/ . [5] Cand ` es E . Compressiv e sampling. In P r oc. International Co ngr ess of Mathemat icians, vol. 3, Madrid, Spain (2006). [6] Cand ` es E., Romberg J. and T ao T ., Robust uncertaint y principl es: exa ct signal rec onstruction from highly inco mplete frequenc y information. Informatio n Theory , IEE E T ransacti ons on, vol. 52, no. 2, pp. 489–509 (2006). [7] Cand ` es E. , and T ao T ., Decoding by linea r programming. IE EE T rans. on Information Theory , 51(12), pp. 4203 - 4215 (2005). [8] Donoho D., Compressed sensing. IEEE Tr ansactions on Info rmation Theory , vol. 52, no. 4, pp. 1289–1306 (2006). [9] Donoho D.L . and Elad M., Optimally sparse representat ion in general (non-orthog onal) dictiona ries vi a l 1 minimiz ation. Proc . Natl . Acad. Sci. USA 100, no. 5, 2197–2202 (2003). [10] DeV ore R. A. , Deterministi c constructi ons of compressed sensing ma- trices. J. Complexi ty 23 (2007), no. 4-6, 918–925. [11] Daubec hies I., Grossmann A . and Meyer Y ., Painless non-orthogona l expa nsions. J . Math. Phys., 27 (5), pp. 1271-1283 (1986). [12] Elad M. and Bruckstein A.M., A Generalized Uncertainty Principl e and Sparse Represent ation in Pairs of Bases. IEEE T rans. On Information Theory , V ol. 48, pp. 2558-2567 (2002). [13] Gure vich S. and Hadani R., The geometric W eil represent ation . Selecta Mathemat ica, New Series, V ol. 13, No. 3. (Dece mber 2007), pp. 465-481. [14] Gure vich S. and Hadani R., T he statistical restricted isometry property and the W igner semicircle distrib ution of incoheren t dictionarie s. Sub- mitted to the Annals of Applied Probabi lity (2009). [15] Gure vich S., Hadani R. and Sochen N., The finite harmonic oscilla tor and its associate d s equenc es. Proce edings of the National Academy of Scienc es of the United States of America, in press (2008). 3 [16] Gure vich S., Hadani R., Sochen N., On s ome determin istic dicti onaries supporting sparsity . Special issue on sparsity , the Journa l of F ourier Analysis and Applicati ons. T o appear (2008). [17] Gure vich S., Hadani R., Sochen N., The finite harmonic oscil lator and its applic ations to sequenc es, communicat ion and radar . IEEE T ransact ions on Information Theory , vol. 54, no. 9, September 2008. [18] Ho we R., Nice error bases, mutually unbiased bases, induce d represe n- tatio ns, the Heisenber g group and finite geometries. Indag . Math. (N.S.) 16 , no. 3-4, 553–583 (2005). [19] Ho ward S. D. , Calderba nk A. R. and Moran W . T he finite Heisenbe rg- W eyl groups in radar and communication s. EURASIP J. Appl. Signal Pr ocess. (2006). [20] Ho ward S.D., Calder bank A.R., and S earle S.J. , A fast reconstruc tion algorit hm for determinist ic compressi ve sensing using second order Reed-Mul ler codes. CISS (2007). [21] Indyk P ., Explicit constructi ons for compressed sensing of sparse signal s. SOD A (2008) . [22] Jafa rpour S., Ef ficient Compressed Se nsing using Lossless Ex- pander Graphs with Fast Bilate ral Quantum Recov ery Algorith m. arXiv:0806.3799 (2008). [23] Jafa rpour S., Xu W . , Hassibi B., Calderba nk R., Efficient and Robust Compressi ve S ensing using High-Quality Expander Graphs. Submitted to the IEEE transac tion on Information Theory (2008). [24] Xu W . and Hassibi B., Efficient Compressi ve Sensing with Deterministi c Guarante es using Expander Graphs. P r oceedings of IE EE Information Theory W orkshop, Lake T ahoe (2007). [25] Saligra m a V ., Deterministic Designs with Determinist ic Guarantees: T oeplit z Compressed Sensing Matrices, Sequence Designs and System Identific ation. arXiv:0806.4958 (2008). [26] Tro pp J.A., On the conditio ning of random subdictio naries. Appl. Comput. Harmonic Anal., vol. 25, pp. 1–24, 2008. [27] Tro pp J .A., Norms of random submatrices and sparse approximation. Submitte d to Comptes-Rendu s de l’Acad ´ emie des Scie nces (2008). [28] W eil A., Sur certains groupes d’operate urs unitaires. A cta Math. 111 , 143-211 (1964).