The statistical restricted isometry property and the Wigner semicircle distribution of incoherent dictionaries

THE ST A TISTICAL RESTRICTED ISOMETR Y PR OPER TY AND THE WIGNER SEMICIR CLE DISTRIBUT ION O F INCOHERENT DICTIONARIES SHAMGAR GUREVICH AND RONNY HAD ANI Abstract. In this article we present a statistical version of the Cand` es-T ao restricted i sometry prop erty (SRIP for short) which holds in general f or any incoheren t dictionary whic h i s a disjoint union of orthonormal bases. In ad- dition, we sho w that, under appropriate normal ization, the eigen v alues of the associated Gram matrix fluctuate around λ = 1 according to the Wigner s emi- circle distribution. The result is then applied to v arious dictionaries that arise naturally in the setting of finite harmonic analysis, giving, in particular, a better understanding on a remark of Applebaum-Ho wa rd-Searle-Cal derbank concerning RIP for the Heisenberg dictionary of c hirp li k e functions. 0. Intr oduction Digital signals , or simply signals, can b e thought of a s co mplex v alued functions on the finite field F p , where p is a prime num be r. The space o f s ig nals H = C ( F p ) is a Hilb ert spa ce o f dimension p , with the inner pro duct g iven b y the sta ndard formula h f , g i = P t ∈ F p f ( t ) g ( t ) . A dictio nary D is simply a set of vectors (a ls o called atoms ) in H . The num b er of vectors in D ca n exceed the dimensio n of the Hilb ert space H , in fact, the most int eresting situation is when | D | ≫ p = dim H . In this set-up we define a r esolut ion of the Hilb ert space H via D , which is the morphism of vector spa c es Θ : C ( D ) → H , given by Θ ( f ) = P ϕ ∈ D f ( ϕ ) ϕ , for every f ∈ C ( D ). A more concrete way to think of the morphism Θ is a s a p × | D | ma trix with the columns b eing the ato ms in D . In the last t wo decades [11], and in particular in recent years [3, 4, 5, 6, 7, 8], resolutions of Hilbe r t space s b ecame an imp orta n t to ol in sig nal pr o cessing, in particular in the emerging theories of spars it y and compres sive sensing . 1. The restricted isometr y pr oper ty A useful pro p er t y of a resolution is the restricted isometry pr op erty (RIP for short) defined by Cand` es-T ao in [7]. Fix a natura l num ber n ∈ N and a pa ir of po sitive r eal num ber s δ 1 , δ 2 ∈ R > 0 . Date : Dec. 1, 2008. c  Cop yright b y S. Gurevic h and R. Hadani, Dec. 1, 2008. All righ ts reserve d. 1 2 SHAMGAR GUREVICH AND RONNY HADANI Definition 1.1. A dictionary D satisfies the r estricte d isometry pr op erty with c o- efficients ( δ 1 , δ 2 , n ) i f for every subset S ⊂ D such that | S | ≤ n we hav e (1 − δ 2 ) k f k ≤ k Θ ( f ) k ≤ (1 + δ 1 ) k f k , for every function f ∈ C ( D ) which is supp orte d on the set S . Equiv a le ntly , RIP can b e form ulated in terms of the spectral radius of the cor- resp onding Gram op era tor. Let G ( S ) denote the comp ositio n Θ ∗ S ◦ Θ S with Θ S denoting the restrictio n of Θ to the subspac e C S ( D ) ⊂ C ( D ) o f functions sup- po rted on the set S . The dictionary D satisfies ( δ 1 , δ 2 , n )-RIP if for every subset S ⊂ D suc h that | S | ≤ n we hav e δ 2 ≤ k G ( S ) − I d S k ≤ δ 1 , where I d S is the identit y o p er ator on C S ( D ). It is k nown [2 , 8] that the RIP holds for r andom dictio naries. How ever, one would like to addres s the following problem [1, 10, 9, 2 0, 2 1, 2 2, 23, 25, 24, 26, 2 7]: Problem 1.2 . Find deterministic c onstru ction of a dictionary D with | D | ≫ p which satisfies RIP with c o efficients in the critic al r e gime (1.1) δ 1 , δ 2 ≪ 1 and n = α · p, for some c onstant 0 < α < 1 . 2. In coherent dictionaries Fix a po sitive rea l num b er µ ∈ R > 0 . The following notion was introduce d in [9, 1 2] a nd was used to study similar problems in [26, 27]: Definition 2.1. A dictionary D is c al le d inc oher ent with c oher enc e c o efficient µ (also c al le d µ - c oher en t) if for every p air of distinct atoms ϕ, φ ∈ D |h ϕ, φ i| ≤ µ √ p . In this a rticle we will explor e a genera l re lation b etw een RIP and incoherence. Our motiv ation comes from three examples of incoheren t dictionaries whic h arise naturally in the setting of finite harmonic analysis: • The first example [18, 19], r eferred to a s the Heisenb er g dictionary D H , is constructed using the Heisen b erg representation of the finite Heisenberg group H ( F p ). The Heisen b erg dictionary is of size approximately p 2 and its coherence coe fficien t is µ = 1. • The second exa mple [15, 16, 17], which is referred to a s the oscil lator dic- tionary D O , is constructed using the W eil repr e sentation of the finite sym- plectic gro up S L 2 ( F p ). The oscillato r dictio nary is of size approximately p 3 and its coher ence co efficie nt is µ = 4. • The third exa mple [15, 1 6, 1 7], r eferred to as the ext ende d oscil lator dictio- nary D E O , is constr uc ted using the Heisenberg-W eil representation [2 8, 13] of the finite Jaco bi group, i.e., the semi-direct pr o duct J ( F p ) = S L 2 ( F p ) ⋉ H ( F p ). The ex tended o scillator dic tio nary is of size approximately p 5 and its coherence coe fficien t is µ = 4. ST A TISTICAL RESTRICTED ISOM ETR Y PR OPER TY 3 The three examples of dictiona ries we just describ ed constitute re a sonable ca n- didates for so lving Problem 1 .2: They are lar g e in the sens e that | D | ≫ p, and empirical evidences sugges t (see [1] for the cas e of D H ) that they might satisfy RIP with c o efficients in the critical regime (1.1). W e s ummarize this as follows: Question: Do the dictionaries D H , D O and D E O satisfy the RIP with co ef- ficient s δ 1 , δ 2 ≪ 1 a nd n = α · p , for some 0 < α < 1? 3. Main resul ts In this ar ticle we formulate a rela xed statis tica l version of RIP , called statistica l isometry pr op erty (SRIP for short) which holds for any incoher ent dictionary D which is, in addition, a disjoint union of orthonorma l bases: (3.1) D = a x ∈ X B x , where B x =  b 1 x , .., b p x  is an or thonormal ba sis o f H , for every x ∈ X . 3.1. The statisti cal isometry prop erty . Let D be an incoheren t dictionar y o f the form (3.1). Roug hly , the statemen t is that for S ⊂ D , | S | = n with n = p 1 − ε , for 0 < ε < 1, chosen uniformly a t random, the op erator norm k G ( S ) − I d S k is small with high probability . P recisely , we hav e Theorem 3 .1 (SRIP pro pe rty [14]) . F or every k ∈ N , ther e exists a c onstant C ( k ) such that t he pr ob ability (3.2) P  k G ( S ) − I d S k ≥ p − ε/ 2  ≤ C ( k ) p 1 − εk/ 2 . The ab ov e theorem, in particular, implies that pr o bability P  k G ( S ) − I d S k ≥ p − ε/ 2  → 0 a s p → ∞ faster then p − l for a ny l ∈ N . 3.2. The statistics of the eigenv alues . A natura l thing to know is how the eigenv alues o f the Gram op erato r G ( S ) fluctuate a round 1 . In this regard, we study the sp ectral statistics of the norma lized err or term E ( S ) = ( p/n ) 1 / 2 ( G ( S ) − I d S ) . Let ρ E ( S ) = n − 1 P n i =1 δ λ i denote the sp ectral distribution of E ( S ) where λ i , i = 1 , .., n , are the rea l eigenv a lues of the Hermitian op era tor E ( S ). The following theorem a s serts that ρ E conv erges in pr obability as p → ∞ to the Wigner semicircle distribution ρ S C ( x ) = (2 π ) − 1 √ 4 − x 2 · 1 [2 , − 2] ( x ) where 1 [2 , − 2] is the c haracter is tic function of the in terv al [ − 2 , 2]. Theorem 3.2 (Semicircle distribution [14]) . We have (3.3) lim p →∞ ρ E P = ρ S C . Remark 3.3. A limit of the form (3.3) is famili ar in r andom matrix t he ory as the asymptotic of the sp e ctr al di stribution of Wigner matric es. Inter estingly, t he same asymptotic distribution app e ars in our situation, alb eit, the pr ob ability sp ac es ar e of a differ ent natur e (our pr ob ability sp ac es ar e, in p articular, mu ch smal ler). 4 SHAMGAR GUREVICH AND RONNY HADANI In par ticular, Theor ems 3.1, 3.2 can be applied to the three examples D H , D O and D E O , which are all o f the appro priate for m (3.1). Finally , o ur result gives new information on a remark of Applebaum-Ho ward-Searle-Calder bank [1 ] concerning RIP o f the Heisenberg dictionary . Remark 3.4. F or pr actic al applic ations, it might b e imp ortant to c ompute ex- plicitly t he c onstants C ( k ) which app e ars in (3.2). This c onstant dep ends on the inc oher en c e c o efficient µ , ther efor e, for a fixe d p , having µ as smal l as p ossible is pr efer able. Ac knowledgement 3.5. It is a ple asur e to thank our te acher J. Bernstein for his c ontinu os supp ort. We ar e gr ateful to N. So chen for many stimulating discussions. We thank F. Bruckstein, R. Calderb ank, M. Elad, Y. Eldar, R. Kimmel, and A. Sahai for sharing with us some of their thoughts ab out signal pr o c essing. We ar e gr ateful to R. Howe, A. Man, M. R evzen and Y. Zak for ex plaining us the notion of mutu al ly u nbiase d b ases. References [1] Applebaum L. , How ard S., Searle S., and Calderbank R., Chirp sensing codes: Deterministic compressed sensing measuremen ts for fast recov ery . (Pr eprint, 2008). [2] Baraniuk R., Dav enp ort M . , DeV ore R.A. and W akin M .B., A simple proof of the r estricted isometry property for random matrices. Constructive Appr oximation, t o app e ar (2007). [3] Bruckstein A.M., Donoho D.L. and Elad M., ”F rom Sparse Solutions of Systems of Equations to Sparse Modeli ng of Signals and Images”, to app ea r in SIAM R eview (2007). [4] Compressive Sensing Resources. Av ailable at http ://www.dsp.ece.rice.edu/cs/ . [5] Cand` es E. Compressi v e sampling. In Pr o c. International Congr ess of Mathematicians, vol. 3, Madrid, Sp ain (2006). [6] Cand` es E., Romberg J. and T ao T., Robust uncertain t y principles: exact signal reconstruction from highly incomplete frequency information. Information The ory, IEEE T r ansactions on, vol. 52, no. 2, pp. 489–509 (2006). [7] Cand` es E., and T ao T., Deco di ng by linear programming. IEEE T r ans. on Information The ory, 51(12), pp. 4203 - 4215 (2005). [8] Donoho D., Compressed sensing. IEEE T r ansactions on Information The ory, vol. 52, no. 4, pp. 1289–1306 (2006). [9] Donoho D.L. and Elad M., Optimally sparse represen tation in general (non-orthogon al) dic- tionaries via l 1 minimization. Pr o c. Natl. A c ad. Sci. USA 100, no. 5, 2197–2202 (2003). [10] DeV ore R. A., Deterministic constructions of compressed sensing matrices. J. Complexity 23 (2007), no. 4-6, 918–925. [11] Daubechies I., Grossmann A. and Meyer Y. , P ainless non-orthogona l expan sions. J . Math. Phys., 27 (5), pp. 1271-1283 (1986). [12] Elad M. and Br uc kstein A.M ., A Generalized Uncertaint y Pr inciple and Sparse Representa - tion in Pairs of Bases. IEEE T r ans. On Information The ory, V ol. 48, pp. 2558-2567 (2002) . [13] Gurevich S. and Hadani R., The geometric W eil r epresen tation . Sele cta Mathematic a, New Series, V ol. 13, No. 3. (De c emb er 2007 ), pp. 465-481. [14] Gurevich S. and Hadani R. , Inco herent dictionaries and the statistical restricted isometry property . Submitte d to IEEE journal of sele c t e d t opics in signal pr o c essing - sp e cial issue on c ompr essive sensing (2008). [15] Gurevich S., Hadani R. and So chen N., T he finite harm onic oscillator and its asso ci ated sequences. Pr o c e ed ings of the National A ca demy of Scie nc e s of the Unite d States of A meric a, in pr ess (2008). [16] Gurevich S., Hadani R., Sochen N., On some deterministic dictionaries s upporting sparsit y . Sp e cial issue on sp arsity , t he Journal of F ourier Ana lysis and Applic ations. T o app e ar (2008). [17] Gurevich S., Hadani R., So chen N. , The finite harmonic oscillator and its app lications to sequences, communicat ion and r adar . IEEE T r ansactions on Information The ory, vol. 54, no. 9, Septemb er 2008. ST A TISTICAL RESTRICTED ISOM ETR Y PR OPER TY 5 [18] How e R., Nice error bases, mut ually un biased bases, induced representat ions, the Heisen ber g group and finite geomet ries. Indag. Math. (N. S.) 16 , no. 3-4, 553–58 3 (2005). [19] How ard S. D., Calderbank A . R. and Moran W. The finite H ei s en b erg-W eyl groups in radar and comm unications. EURASIP J. Appl. Signal Pr o c ess. (2006). [20] How ard S. D ., Cal derbank A. R., and Searle S.J., A fast reconstruction al gori thm f or deter- ministic compressive sensing using second order Reed-Muller co des. CISS (2007). [21] Indyk P ., Explicit constructions for compressed sensing of sparse signals. SODA (2008) . [22] Jafarp our S. , Efficien t Compr essed Sensing using Lossless Expander Graphs with F ast Bilat- eral Quan tum Recov ery Algorithm. arXiv:0806.3799 (2008). [23] Jafarp our S., Xu W., Hassi bi B., Calderbank R., Efficient and Robust Compressive Sensing using High-Q uali t y Expande r Graphs. Submitte d to the IEEE tr ansaction on Information The ory (2008). [24] Xu W. and Hassibi B., Efficient Compressive Sensing with Deterministic Guarant ees using Expander Gr aphs. Pr o c e e dings of IEEE Information The ory Workshop, La ke T aho e (2007). [25] Saligrama V., Deterministic Designs with Deterministic Guaran tees: T o eplitz Compr essed Sensing Matrices, Sequenc e Designs and System Iden tification. arXiv : 0806.49 58 (2008). [26] T ropp J.A., On the condit ioning of random s ub dictionaries. Appl. Comput. Harmonic Anal ., vol. 25, pp. 1–24, 2008. [27] T ropp J. A., Nor ms of random submatrices and sparse approximation. Submitt e d to Comptes- R endus de l’A c ad ´ emie des Scienc es (2008). [28] W eil A., Sur certains groupes d’ operateurs unitaires. A cta Math. 111 , 143-211 (1964). Dep ar tment of Ma thema tics, University of California, Berkeley, CA 9472 0, USA. E-mail addr ess : shamgar@math. berkeley.edu Dep ar tment of Ma thema tics, University of Chicago, IL 60637 , USA. E-mail addr ess : hadani@math.u chicago.edu