Lower Bounds on the Rate-Distortion Function of Individual LDGM Codes

Lo wer Bounds on the Rate-Distortion Fun ction of Indi vidual LDGM Codes Shrini vas Kudekar and R ¨ udiger Urbanke EPFL, School of Computer and Communication Sciences, Lau sanne 101 5, Switzerland Abstract — W e consider lossy compression of a binary symmet- ric sour ce by means of a low-density generator -matrix code. W e derive two lower bound s on the rate distortion function which are valid for any lo w-density generator -matrix code wit h a given node d egree distribution L ( x ) on the set of generators and for any encoding algorithm. These bounds show that, due to the sparseness of the code, t he p erf ormance is stri ctly bounded away from the Sh annon rate-distortion function. In this sense, our bounds represent a natural generalization of Gallager’s bound on the maximum rate at which low-density p arity-check codes can be used for r eliabl e transmission. Our bounds are similar in spirit to the technique recently d ev el oped by Dimakis, W ainw right, and Ramchandran, but they apply to individual codes. I . I N T RO D U C T I O N W e consider lossy compression of a bin ary symm etric source (BSS) using a low-density gene rator-matrix (LDGM) code as shown in Figure 1. More precisely , let S ∈ F m 2 represent the bina ry sou rce of length m . W e have S = { S 1 , S 2 , . . . , S m } , wh ere the { S i } m i =1 are iid rand om variables with P { S i = 1 } = 1 2 , i ∈ [ m ] . Let S denote the set of all source word s. ˆ S 1 ˆ S 2 ˆ S 3 ˆ S 4 ˆ S 5 ˆ S 6 ˆ S 7 S 1 S 2 S 3 S 4 S 5 S 6 S 7 W 1 W 2 W 3 W 4 Fig. 1. The T anner graph correspo nding to a simple LDGM co de used for lossy compression of a BSS. W e have m = 7 , R = 4 7 , and L ( x ) = x 3 . Giv en a sour ce w o rd s ∈ S , we co mpress it by mapping it to one o f the 2 mR index words w ∈ W = F mR 2 , wh ere R is the rate , R ∈ [0 , 1 ] . W e denote this encod ing map by f : s 7→ W (the map can be random ). The recon struction is do ne via an LDGM co de deter mined by a sparse b inary mR × m gener ator matrix G . Let ˆ s denote the reconstru cted word associated to w . W e have ˆ s = w G . W e denote this decod ing map by g : w 7→ ˆ s . Let ˆ S deno te th e code, ˆ S = { ˆ s (1) , . . . , ˆ s (2 mR ) } , ˆ s ( i ) ∈ F m 2 . The co dew o rds are n ot necessarily distinct. W e call the compo nents of th e ind ex word w = { w 1 , . . . , w mR } the generators and the associated n odes in th e factor graph representing t he LDGM code the generator n odes . W e assume that th ese gen erators nod es have a normalized degree distribution L ( x ) = P i L i x i . This means that L i represents the fraction (o ut of mR ) of generator nod es of degree i . W e are interested in the trade-off b etween rate and distortion which is achiev ab le in th is setting. Let d( · , · ) denote the Hamming distortion fu nction, d : F m 2 × F m 2 → N . The average distortion is the n given by 1 m E [ d ( S, g(f ( S ))] . W e are interested in th e min imum o f th is a vera ge distortio n, where the minimum is taken over all LDGM co des o f a giv en rate, generato r degree distribution L ( x ) , and length , as well as over all encodin g functio ns. I I . R E V I E W Giv en the success of sparse gr aph codes ap plied to the channel co ding pro blem, it is not surprising tha t ther e is also interest in the use o f sparse graph codes f or the source coding problem . Martinian and Y edidia [1] were probably the first to work on lossy compr ession using spa rse gr aph cod es. Th ey considered a mem oryless ternar y sour ce with erasures and demonstra ted a duality result between c ompression of this source an d the transmission prob lem over a binar y erasure channel (bo th using iterative en coding/d ecoding ). Mezard, Zecchina, and Ciliberti [2] considered the lo ssy co mpression of the BSS using LDGM co des with a Poisson d istribu- tion on the gene rators. They derived the o ne-step replica symmetry- breaking (1RSB) solution and the average rate- distortion fun ction. Accor ding to this ana lysis, this ensemble approa ches the Shan non rate-distortion curve expon entially fast in the a verage degree. They observed that the iterative interpretatio n associated to the 1RSB analy sis giv es rise to an algor ithm, which they ca lled survey pr opagation . In [3] the same author s implement an encoder that utilizes a T an ner graph with r andom non-linear functions at the check nodes and a surve y pr o pagation based d ecimation algo rithm for d ata compression of the BSS. In [4], W ainwright and Mane va also considered th e lo ssy com pression o f a BSS using an LDGM code with a given degree distribution. They sho wed how survey pro pagation can be interp reted as belief propagation algorithm (a s did Braun stein an d Z ecchina [ 5]) o n a n en larged set o f assign ments an d d emonstrated that the sur vey p ropa- gation algorithm is a practical and efficient encoding scheme. Recently , Filler and Fried rich [6] demo nstrated experimenta lly that even stan dard b elief pro pagation based decimation algo- rithms usin g optim ized degree d istributions for LDGM cod es and a pr oper in itialization of the messages c an ac hiev e a rate- distortion trade-off very close to the Shannon bound. Martinian and W ainwright [7], [8], [9] constru cted co mpound LDPC and LDGM code ensembles an d gave r igorou s up per bou nds on their distortion perf ormance. A standar d L DGM code ensemble is a special case of their construction, hence they also provide upper bounds on the rate-distortio n function of LDGM ensembles. By using the first and second momen t method they proved that a code chosen rand omly from the compound ensemble under optimal encodin g and d ecoding achieves th e Shannon rate-distortion curve with high probability . Finally , they po inted out that such con structions are u seful also in a m ore gener al context (e.g., the W yner-Ziv o r the Gelfand- Pinsker problem) . Dim akis et al [1 0] were the first autho rs to provide rigoro us lower bo unds on the r ate-distortion fun ction of LDGM co de ensembles. Theor e m 1 (Dimakis, W ainwright, Ra mchandran [10] ): Let ˆ S be a binar y code of blocklength m and rate R chosen unifor mly at rando m from an en semble of lef t Poisson LDGM Codes with chec k-nod e d egree r . Suppo se that we p erform MAP decod ing. W ith high pro bability th e r ate-distortion pair ( R, D ) ach ie ved by ˆ S fulfills R ≥ 1 − h ( D ) 1 − e − (1 − D ) r R > 1 − h ( D ) . A. Outline In the spirit of Gallager ’ s inform ation theoretic bou nd for LDPC cod es, we are interested in d eriving lower bound s on the rate-d istortion function which are valid for any LDGM code with a given generator nod e degree distribution L ( x ) . Our ap proach is very simple. Pick a parame ter D , D ∈ [0 , 1 2 ] (think of this parameter as the distortion) . Con sider the set of “covered” sequences C ( D ) = [ ˆ s ∈ ˆ S B ( ˆ s, D m ) , (1) where B ( x, i ) , x ∈ F m 2 , i ∈ [ m ] , is th e Hamming b all of rad ius i centered at x . I n words, C ( D ) represents the set of all those source seq uences th at ar e within Hamming distance at most D m f rom at least one code word. Recall that fo r any s ∈ S , f ( s ) ∈ W r epresents the index word and t hat g(f ( s )) denotes the reconstructed word. W e have d( s, g (f ( s ))) ≥ ( 0 , s ∈ C ( D ) , D m, s ∈ F m 2 \ C ( D ) . Therefo re, 1 m E [d( S, g (f ( S )))] = 1 m X s ∈ F m 2 2 − m d( s, g (f ( s ))) ≥ 2 − m m X s ∈ F m 2 \C ( D ) d( s, g (f ( s ))) ≥ 2 − m D | F m 2 \ C ( D ) | ≥ D  1 − 2 − m |C ( D ) |  . (2) If th e cod ew or ds are well spread o ut the n we know from Shannon ’ s rand om c oding argument that for a choice D = h − 1 (1 − R ) , |C ( D ) | ≈ 2 m , [11]. But the codewords of an LDGM code are clustered since ch anging a sing le generator symbol only chang es a constant number of sym bols in the codeword. There is theref ore substantial overlap of the balls. W e will sho w that there exists a D which is strictly larger than th e distortion correspo nding to Shan non’ s rate-d istortion bound so th at |C ( D ) | is exponen tially small co mpared to 2 m regardless of th e specific co de. Fro m (2) this im plies that th e distortion is at least D . T o derive the requ ired u pper bound on |C ( D ) | we use two different techniques. In Section III we u se a simple co mbina- torial argument. In Section IV, on the other hand , we employ a probab ilistic argu ment based on the “test channel” which is typically used to show the achievability of the Shan non r ate- distortion fu nction. Although both boun ds prove that the rate-distor tion function is strictly boun ded away from th e Sh annon r ate-distortion function f or the who le rang e of rates an d any LDGM co de, we conjectu re that a stro nger bound is valid. W e pose our conjecture as an open prob lem in Section V. I I I . B O U N D V I A C O U N T I N G Theor e m 2 ( Bound V ia Cou nting): Let ˆ S be an LDGM code with bloc klength m and with generator node degree distribution L ( x ) and d efine L ′ = L ′ (1) . L et f ( x ) = d Y i =0 (1 + x i ) L i , a ( x ) = d Y i =0 iL i x i 1 + x i , ˆ R ( x ) = 1 − h ( x 1+ x ) 1 − log f ( x ) x a ( x ) , ˆ D ( x ) = x 1 + x − a ( x ) ˆ R ( x ) . For R ∈ [ 1 L ′ , 1] let x ( R ) be the u nique p ositiv e solutio n of ˆ R ( x ) = R . Define the cu rve D ( R ) a s    1 2  1 − RL ′  1 − 2  x ( 1 L ′ ) 1+ x ( 1 L ′ ) − a ( x ( 1 L ′ )) l   , R ∈ [0 , 1 L ′ ] , ˆ D ( x ( R )) , R ∈ [ 1 L ′ , 1] . Then, for any block length m , the achievable d istortion of an LDGM c ode of rate R and generator degree distrib ution L ( x ) is lower bou nded by D ( R ) . Discussion: (i) As stated above, if we are considering a single code of rate R then the lower bound o n the distor tion is D ( R ) . If, on th e other hand w e are considerin g a family of cod es, all with th e same gene rator degree distribution L ( x ) but with different rates R , then it is mo re conv enient to plot the lower bound in a parametric f orm. First p lot th e curve ( ˆ D ( x ) , ˆ R ( x )) for x ∈ [0 , 1 ] . Th en conn ect the p oint ( D = 1 2 , R = 0) to the point o n the ( ˆ D ( x ) , ˆ R ( x )) curve with ˆ R ( x ) = 1 L ′ by a straight line. The resultin g upper en velope gi ves the stated lower bound fo r the whole range. This con struction is shown in Figure 2 . (ii) Although this is difficult to glance from th e expression s, we will see in the proof th at for any bound ed gen erator degree distribution L ( x ) th e perfo rmance is strictly boun ded away from th e Sh annon r ate-distortion function . Fro m a practical persp ectiv e howe ver the gap to the rate-distortio n bound decr eases quick ly in the degree. 0 . 1 0 . 2 0 . 3 0 . 4 0 . 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 R D 1 L ′ = 1 2 1 ( L ′ ) 2 = 1 4 ˆ D ( x ( 1 L ′ )) ≈ 0 . 115 L ′ − 1 2 L ′ = 1 4 bound on achiev able region Fig. 2. Constructi on of the bound for code s with L ( x ) = x 2 so that L ′ = 2 (all generator nodes hav e degree 2 ). The solid gray curve corresponds to the Shannon rate-distort ion curve. The black curve just abov e, which is parti ally solid and partially dotted, corresponds to the curve ( ˆ D ( x ) , ˆ R ( x )) for x ∈ [0 , 1] . It start s at the point (0 , 1) (which corresponds to x = 0 ) and ends at ( L ′ − 1 2 L ′ = 1 4 , 1 ( L ′ ) 2 = 1 4 ) which corresponds to x = 1 . The straight line goes from the point ( ˆ D ( x ( 1 L ′ )) , 1 L ′ ) to the point ( 1 2 , 0) . Any achie vabl e ( R, D ) pair must lie in the lightly shaded region. This regio n is strictly bounded aw ay from the Shannon rate-disto rtion funct ion ov er the whole range. Example 1 (Gen erator - Re g ular LDGM Codes): Consider codes with gener ator degree equal to l and an arbitrar y degree distribution on the check n odes. In this case we have f ( x ) = 1 + x l and a ( x ) = l x l 1+ x l . Figu re 3 compar es the lower b ound to the rate- distortion cur ve fo r l = 1 , 2 , an d 3 . For each case the achiev able region is strictly bou nded away from the Shan non rate- distortion curve. 0 . 1 0 . 2 0 . 3 0 . 4 0 . 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 R D l = 1 l = 2 l = 3 Fig. 3. Bound s for L ( x ) = x l for l = 1 , 2 , and 3 . For l = 2 the 3 gray dots correspond to the s pecia l cases R = 2 3 , R = 1 2 , and R = 2 5 respect i vely . The correspondi ng lo wer bounds on the distortion are D ( 2 3 ) ≥ 0 . 0616 > 0 . 0614905 (rate-distor tion bound), D ( 1 2 ) ≥ 0 . 115 > 0 . 11 (rate-distort ion bound), and D ( 2 5 ) ≥ 0 . 1924 > 0 . 1461 (rate-distorti on bound). Example 2 ( ( l , r ) -Regular LDGM Codes): In th is case we have R = l / r and L ( x ) = x l . T he same bound as in Example 1 applies. The three spe cial cases ( l = 2 , r = 3) , ( l = 2 , r = 4) , a nd ( l = 2 , r = 5) , which correspon d to R = 2 3 , R = 1 2 , and R = 2 5 respectively , are marked in Figure 3 as gr ay dots. Example 3 ( r - Re g ular LDGM Cod es of Rate R ): Assume that all check no des have d egree r and that the con nections are chosen u niformly at r andom with repetitio ns. For large blockleng ths th is implies that the de gree distribution on the variable nodes conver g es to a Poisson distribution, i.e., we have in the limit L ( x ) = ∞ X i =1 L i x i = e r R ( x − 1) . Let u s e valuate o ur bound fo r th is generator degree distribu- tion. Note that since the average d egree of the check node s is fixed we have a different generator d egree distribution L ( x ) for eac h rate R . Figure 4 compares th e resulting bound with the Shan non rate-distortion functio n as well as the boun d of Theo rem 1. The ne w bo und is sligh tly tighter . But more importan tly , it applies to a ny LDGM code. 0 . 025 0 . 05 0 . 075 0 . 100 0 . 125 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 0 . 9 R D r = 2 r = 4 Fig. 4. Lo wer bound on achie vab le ( R, D ) pairs for r -re gular LDGM codes with a Poisson generator degree distribut ion and r = 2 , 4 . The dashed curve correspond s to the bound of Theorem 1 and the solid black curve represents the bound of Theorem 2. The gray curve is the Shannon rat e-distortion tradeof f. Pr oof of Theo r em 2. From the statement in Theorem 2 you see that the bound co nsists of a portion of the cu rve ( ˆ D ( x ) , ˆ R ( x )) an d a stra ight-line portion . The straight-lin e portion is easily explained. Assum e that all generato r nodes have degree l (for the general case replace all m entions of l by the average degree L ′ ). The n th e maxim um nu mber of check nod es that can depend on the choice o f gener ator no des is n l . Therefo re, if the rate R is lower than 1 l then at least a fr action (1 − R l ) of th e ch eck no des canno t be connected to any gen erator node. For those no des the average distortion is 1 2 , wh ereas for the fraction R l of the ch eck nodes which are (potentially ) con nected to a t least one gen erator node the best ach iev able distor tion is the same for any 0 ≤ R ≤ 1 l . I t suffices th erefore to restrict our attention to rates in the range [ 1 L ′ , 1] and to prove that their ( R , D ) pairs are lower b ounde d by th e curve ( ˆ D ( x ) , ˆ R ( x )) . As a second simp lification n ote that althou gh the b ound is valid for all blockleng ths m we only n eed to pr ove it fo r the limit of infinite block lengths. T o see th is, co nsider a particu lar code of blo cklength m . T ake k identical copies of this code and co nsider the se k cop ies as one code of block length k m . Clearly , this lar g e code has the same rate R , the s ame generator degree distribution L ( x ) , and the same distortion D as each compon ent code. By letting k tend to infinity we can constru ct an arbitrarily large code of the sam e characteristics and app ly the b ound to this limit. Since our bou nd below is v alid f or any sequ ence of co des whose blocklength ten ds to infinity the claim follows. Pick w ∈ N so that D m + w ≤ m 2 . T hen |C ( D ) | = | [ ˆ s ∈ ˆ S B ( ˆ s, D m ) | (i) ≤ 1 A m ( w ) X ˆ s ∈ ˆ S |B ( ˆ s, D m + w ) | (ii) ≤ 2 − mR log f ( x ω ) x ω ω + o m (1) 2 mR 2 mh ( D + w/m ) (iii) = 2 m ( − R log f ( x ω ) x a ( x ω ) ω + R + h ( D + a ( x ω ) R )+ o m (1)) . T o see (i) note that a “b ig” sphere B (ˆ s, D m + w ) , where ˆ s ∈ ˆ S , contains a ll “small” spher es of the form B ( ˆ s ′ , D m ) , where ˆ s ′ ∈ ˆ S so that d( ˆ s, ˆ s ′ ) ≤ w . L et A m ( w ) be the number of codew ords of Ha mming weight at mo st w . T hen, by sym metry , each small sp here B (ˆ s ′ , D m ) is in exactly A m ( w ) big spheres B ( ˆ s, D m + w ) . It f ollows th at every po int in S ˆ s ∈ ˆ S B ( ˆ s, D m ) is co unted at least A m ( w ) times in the expression P ˆ s ∈ ˆ S |B ( ˆ s, D m + w ) | . Consider no w step (ii). W e ne ed a lower bo und o n A m ( w ) . Assume at first that all g enerator nodes have degree l . Assume that exactly g gener ator nodes are set to 1 and that all other nodes are set to 0 . There are  mR g  ways of do ing this. No w note that for each such co nstellation the weight of the resultin g codeword is at most w = g l . It f ollows that in the ge nerator regular case we have A m ( w ) ≥ w/ l X g =0  mR g  . (3) W e can rewrite (3) in the form A m ( w ) ≥ w X i =0 coef { (1 + x l ) mR , x i } , (4) where coef { (1 + x l ) mR , x i } in dicates the coefficient of th e polyno mial (1 + x l ) mR in fron t of th e monom ial x i . The expression (4) stays valid also for irregular generator degree distributions L ( x ) if we r eplace (1 + x l ) mR with f ( x ) mR , where f ( x ) = Q i (1 + x i ) L i as defin ed in th e statemen t of th e theorem. This of course requir es that n is cho sen in suc h a way that nL i ∈ N f or all i . Define N m ( w ) = P w i =0 coef { f ( x ) mR , x i } , so that (4) can be restated as A m ( w ) ≥ N m ( w ) . Step (ii) no w follows by using the asymptotic e xp ansion of N m ( w ) stated as Theor em 1 [12], where we define ω = w / ( mR ) and wh ere x ω is the unique p ositi ve solution to a ( x ) = ω . Finally , to see (iii) we replace w by mRa ( x ω ) and thus we get the claim. Since this bound is valid fo r any w ∈ N so that D m + w ≤ m 2 we g et the bou nd lim m →∞ 1 m log |C ( D ) | ≤ g ( D , R ) , where g ( D , R ) = inf x ≥ 0 D + a ( x ) R ≤ 1 2 − R lo g f ( x ) x a ( x ) + R + h ( D + a ( x ) R ) . Now n ote that as lon g as g ( D , R ) < 1 , |C ( D ) | is expon en- tially sma ll com pared to 2 m . T herefor e, looking b ack at (2) we see that in th is case the average d istortion conv erges to at least D in the limit m → ∞ . W e g et the tightest bo und by looking f or the condition f or equality , i.e. by lookin g at the equation g ( R, D ) = 1 . If we take the der iv ative with respec t to x and set it to 0 th en we get the condition x 1 + x = D + Ra ( x ) . Recall that D + a ( x ) R ≤ 1 2 , so that this translates to x ≤ 1 . This m eans that x ≤ 1 . Replace D + a ( x ) R in the entro py term by x 1+ x , set the resulting e xpression f or g ( R, x ) equal to 1 , and solve for R . This gi ves R as a fun ction of x and so we also g et D as a fun ction of x . W e have R ( x ) = 1 − h ( x 1+ x ) 1 − log f ( x ) x a ( x ) , D ( x ) = x 1 + x − a ( x ) R ( x ) . A check shows th at x = 0 co rrespond s to ( D , R ) = (0 , 1) and that x = 1 co rrespon ds to ( D , R ) = ( L ′ − 1 2 L ′ , 1 ( L ′ ) 2 ) . Furth er , R and D are monoton e function s of x . Recall that we are only interested in the b ound for R ∈ [ 1 L ′ , 1] . W e get the correspo nding curve by letting x ta ke values in [0 , x ( 1 L ′ )] . For smaller v alues of the rate we get the aforemen tioned straight- line bo und. Lookin g at the above expression for g ( D , R ) on e can see why this boun d is strictly better than the rate-distor tion curve for D ∈ (0 , 1 2 ) . A ssume at first that the gener ator degree distribution is regular . Let the d egree be l . I n th is case a quick check shows that − R log f ( x ) x a ( x ) is equal to − Rh ( a ( x ) l ) . Since a (0) = 0 we g et th e rate distortion bo und if we set x = 0 . Th e claim f ollows by observin g that a ( x ) is a continuo us strictly increasing fun ction and th at h ( x ) has an infinite der i vati ve at x = 0 wh ile h ( D + a ( x ) R ) has a finite d eriv ative a t x = 0 . I t follows that there e x ists a sufficiently small x so that Rh ( a ( x ) l ) is strictly larger than h ( D + a ( x ) R ) − h ( D ) and so th at D + a ( x ) R ≤ 1 2 . Hence, g ( D, R ) is strictly decreasing as a function of x a t x = 0 . T his bou nds the achiev ab le distortion strictly away from the r ate-distortion bound . The same argum ent applies to a n irregular g enerator degree distribution; the simplest way to see this is to replace l by the maxim um degree of L ( x ) . I V . B O U N D V I A T E S T C H A N N E L Instead of using a co mbinato rial appro ach to b ound |C ( D ) | one ca n also use a pro babilistic argument using the “test channel” shown in Figure 5. For the cases we have checked the r esulting b ound is numerically identical to the bou nd o f Theorem 2 (exclu d- ing the straigh t-line por tion). W e restrict our exposition to the regular ca se. The generalization to th e irregular case is straightfor ward. Theor e m 3 ( Bound V ia T est Chan nel): Let ˆ S be an LDGM code with blocklength m , generator de gree distribution L ( x ) = x l , an d rate R . Then f or any p air ( R , D ) , where W 1 W 2 W 3 W 4 ˆ S 1 ˆ S 2 ˆ S 3 ˆ S 4 ˆ S 5 ˆ S 6 ˆ S 7 BSC BSC BSC BSC BSC BSC BSC S 1 S 2 S 3 S 4 S 5 S 6 S 7 Fig. 5. The generat or words W are chosen unifor mly a t random from W . This generates a code word ˆ S uniformly at ra ndom. Each component of ˆ S is then sent over a binary symmetric channel with transition probability D ′ . D is th e average distortion, we have R ≥ sup D ≤ D ′ ≤ 1 2 1 − h ( D ) − KL ( D k D ′ ) 1 − log 2  1 + ( D ′ ) l (1 − D ′ ) l  ≥ 1 − h ( D ) 1 − log 2  1 + D l (1 − D ) l  > 1 − h ( D ) , where KL ( D k D ′ ) = D lo g 2 ( D /D ′ ) + (1 − D ) log 2 ((1 − D ) / (1 − D ′ )) . Pr oof. T he same r emark a s in th e proo f o f Theor em 2 applies: although the b ound is valid for any block length it suffices to prove it for the lim it of b locklength s te nding to infinity . Also, for simp licity we ha ve n ot stated th e bo und in its stre ngthened form which inclu des a straight- line portion. But the same technique that was ap plied in the pro of of Th eorem 2 applies also to the presen t case. As rem arked earlier , the idea of the proof is ba sed on bound ing |C ( D ) | by u sing the “test chan nel. ” More p recisely , choose W unifor mly at ran dom from the set of all b inary sequences o f length mR . Subsequen tly compu te ˆ S via ˆ S = W G , where G is the gene rator matrix of the LDGM co de. Finally , let S = ˆ S + Z , wh ere Z has iid com ponents with P { Z i = 1 } = D ′ . Consider the set of sequen ces s ∈ C ( D ) . For each such s we k now that there exists an ˆ s ∈ ˆ S so that d( s, ˆ s ) ≤ D m . W e have P { S = s | s ∈ C ( D ) } = X ˆ s ′ ∈ ˆ S P { S = s, ˆ S = ˆ s ′ | s ∈ C ( D ) } = m X w =0 X ˆ s ′ ∈ ˆ S : d( ˆ s ′ , ˆ s )= w P { S = s, ˆ S = ˆ s ′ | s ∈ C ( D ) } = m X w =0 A m ( w ) P { S = s, ˆ S = ˆ s ′ | s ∈ C ( D ) , d( ˆ s ′ , ˆ s ) = w } = m X w =0 A m ( w )2 − mR  D ′ 1 − D ′  d( s, ˆ s ′ ) (1 − D ′ ) m ≥ m X w =0 A m ( w )2 − mR  D ′ 1 − D ′  d( s, ˆ s )+d( ˆ s, ˆ s ′ ) (1 − D ′ ) m d( ˆ s ′ , ˆ s )= w = m X w =0 A m ( w )2 − mR  D ′ 1 − D ′  d( s, ˆ s )+ w (1 − D ′ ) m d( s, ˆ s ) ≤ D m ≥ m X w =0 A m ( w )2 − mR  D ′ 1 − D ′  Dm + w (1 − D ′ ) m = 2 − mR − mh ( D ) − m KL ( D k D ′ ) m X w =0 A m ( w )  D ′ 1 − D ′  w , where A m ( w ) deno tes the n umber of codew o rds in ˆ S of Hamming weight w . Due to the linearity of the code this is also the numbe r of co dew o rds in ˆ S of Hamm ing distance w from ˆ s . Using summ ation by parts and setting c = D ′ / (1 − D ′ ) < 1 , we h av e m X w =0 A m ( w ) c w = c m +1 2 mR + m X w =0  w − 1 X i =0 A m ( i )  ( c w − c w +1 ) ( 4 ) ≥ c m +1 2 mR + m X w =0  ⌊ ( w − 1) / l ⌋ X i =0  mR i   ( c w − c w +1 ) = ⌊ m/ l ⌋ X w =0  mR w  c l w + c m +1  2 mR − ⌊ m/ l ⌋ X i =0  mR i   ≥ ⌊ m/ l ⌋ X w =0  mR w  c l w ≥ 1 m (1 + c l ) mR . The last step is valid as lo ng as Rc l 1+ c l < 1 l . I n this case the maximum term (which appears at Rc l 1+ c l m ) is included in the sum (which goes to m/ l ) an d is thus gre ater tha n eq ual to the av e rage of all the terms, which is 1 m (1 + c l ) mR . This condition is trivially fulfilled for R l < 1 . Assume for a moment that it is also fu lfilled for R l ≥ 1 an d the op timum choice of D ′ . It then f ollows that P { S = s | s ∈ C ( D ) } ≥ 1 m 2 − m ( R + h ( D )+ KL ( D k D ′ ) − R log 2 (1+ c l )) . Since 1 = X s ∈ F m 2 P { S = s } ≥ X s ∈C ( D ) P { S = s } ≥ |C ( D ) | 1 m 2 − m ( R + h ( D )+ KL ( D k D ′ ) − R log 2 (1+ c l )) , we have |C ( D ) | ≤ m 2 m ( R + h ( D )+ KL ( D k D ′ ) − R log 2 (1+ c l )) . Pr o- ceeding as in (2), we h av e E [d( S, g (f ( S )))] ≥ D  1 − 2 − m |C ( D ) |  ≥ D  1 − m 2 m ( R + h ( D )+ KL ( D k D ′ ) − R log 2 (1+ c l ) − 1)  . W e conclude th at if for some D ≤ D ′ ≤ 1 2 , R + h ( D ) + KL ( D k D ′ ) − R log 2 (1 + ( D ′ ) l (1 − D ′ ) l ) − 1 < 0 then the distortion is at least D . All this is still co nditioned on R l c l 1+ c l < 1 for the optimum choice of D ′ . For R l < 1 we alread y checked this. So assum e that R l ≥ 1 . The above condition can then equiv ale ntly be wr itten as D ′ < 1 1+( R l − 1) 1 l . On th e other hand, taking the deriv ative of our final expression on the rate- distortion fun ction with respect to D ′ we get the condition f or the maximum to be D ′ = 1 1+(1+ R l D ′ − D ) 1 l < 1 1+( R l − 1) 1 l . W e see therefo re that our assumption R l c l 1+ c l < 1 is also corr ect in the case R l ≥ 1 . Numerical experiments sh ow that the pr esent bou nd y ields for the regular case identical results as plotting the curve correspo nding to g ( D , R ) = 1 , wher e g ( D, R ) was defined in th e proo f of Theorem 2. This can b e interpre ted a s follows. Choose D ′ equal to th e optimal rad ius o f th e Hamming ball in the pro of of Theore m 2. Th en the points ˆ s ′ that con tribute most to the prob ability of S = s must b e th ose that have a distance to ˆ s of m ( D ′ − D ) . V . D I S C U S S I O N A N D O P E N Q U E S T I O N S In the prece ding sections we gave two bounds. Both of them are based o n the idea of co unting the n umber of poin ts that are “c overed” by sph eres ce ntered ar ound the cod ew or ds of an LDGM cod e. In the first case we derived a bou nd b y doub le counting this n umber . In the secon d case we d erived a b ound by lo oking at a prob abilistic model using th e test channel. An intere sting open qu estion is to d etermine the exact relationship of the test chan nel model to the rate-distortio n problem . More precisely , it is tempting to conjecture that a pair ( R, D ) is on ly achiev ab le if H ( S ) = m in this test channel model. This would require to show th at o nly elements of the typical set of S u nder the test cha nnel model are covered, i.e., have code words within distan ce D . For th e test chan nel model it is very easy to determ ine a criterion in the sp irit of Gallager’ s or iginal bo und. W e have H ( S ) = H ( W ) + H ( S | W ) − H ( W | S ) = mR + mh ( D ) − mR X g =1 H ( W g | S, W 1 , . . . , W g − 1 ) (i) ≤ mR + mh ( D ) − mR X g =1 H ( W g | S, W ∼ g ) (ii) = mR + mh ( D ) − mR X g =1 H ( W g | S g , W ∼ g ) , where S g denotes the subset o f the compone nts of the S vectors which are connected to the generator g . Step (i) follows since cond itioning decr eases entropy . Step (ii) fo llows since knowing ( S g , W ∼ g ), W g is not d ependen t o n S ∼ g . Th e ter m H ( W g | S g , W ∼ g ) rep resents the EXIT function of a repetition code whe n tr ansmitting over BSC( D ) channel. If o ne could show that H ( S ) = m is a necessary conditio n for achieving av e rage distortion of D t hen a q uick calculation shows th at the resultin g bo und would read R ≥ 1 − h ( D ) 1 − P l i =0  l i  (1 − D ) i D l − i log 2  1 +  D 1 − D  2 i − l  . This “boun d” is similar in spirit to th e original boun d g iv en by Gallag er , except that in Gallag er’ s o riginal b ound fo r LDPC cod es we have a term corresp onding to the entro py of single- parity check co des, where as here we hav e terms th at correspo nd to the entr opy o f repetition co des; this would be quite fitting given the du ality of the p roblems. A C K N OW L E D G M E N T W e gratefully ack nowledge th e support by the Swiss National Science Foundation under grant number 20 0020 - 11341 2. R E F E R E N C E S [1] E. Martinian and J. Y edidi a, “Iterati ve quanti zatio n using c odes on graphs, ” in Pr oc. of the Allerton Conf . on Commun., Contro l, and Computing , Oct. 2003. [2] S. Cili berti and M. Mezard, “The t heoreti cal capacity of the parity source coder , ” J. Stat. Mech. , 2005. [3] S. Cilibert i, M. Mezard, and R. Zecchina, “Lossy data compression with random gates, ” Phys. Rev . Lett. , vol. 95, 2005. [4] M. J. W ainwright and E . Manev a, “Lossy source co ding via m essage- passing and dec imation o ver gen eraliz ed code words of LDGM codes, ” in Pro c. of the IEEE Int. Symposium on Inform. T heory , Adelaide , Australia , Sept. 2005, pp. 1493–1497. [5] A. Braunst ein and R. Zecchina , “Survey propaga tion as local equil ibrium equati ons, ” J. Statist ical Mechanics: Theory and Experiment , June 2004. [6] T . Filler and J. Fridrich, “Bina ry quantizati on using bel ief propagati on with decimati on over fact or graphs of ldgm codes, ” in Proc. of the Allerton Conf. on Commun., Contr ol, and Computing , Sept. 2007. [7] E. Martinian and M. J. W ainwright, “Lo w-density codes achie ve the rate-di s tortion bound, ” in Proc. of the Data Compre ss ion Confere nce , Sno wbird, UT , Mar . 200 6. [8] ——, “Low-de nsity constructions can achie ve the Wyner-Zi v and Gelfa nd-Pinsker bounds, ” in Proc. of the IEEE Int. Sympo sium on Inform. Theory , Seattle, W A, USA, July 2006, pp. 484–488. [9] ——, “ Analysis of L DGM and compound codes for lossy compression and binning, ” in P r oc. of the IEE E Inform. Theory W orkshop , San Diego, CA, USA, Feb. 2006. [10] A. Dimakis, M. W ainwright, and K. Ramchandran, “Lo w er bounds on the rate-dist ortion function of ldgm codes, ” in Proc . of the IEEE Inform. Theory W orkshop , 2007. [11] T . M. Cov er and J. A. Thomas, Elements of Information Theory . Ne w Y ork, NY , USA: W iley , 1991. [12] D. Burshtein and G. Mill er, “ Asymptotic enumera tion methods for analyz ing LDPC codes, ” IEEE Tr ans. Inform. Theory , vol. 50, no. 6, pp. 1115–1131, June 2004.