A Binary Wyner-Ziv Code Design Based on Compound LDGM-LDPC Structures

A Bina ry W yner-Ziv Co de De si gn Based on Comp ound L DGM- LDPC Structures Mahdi Nangir 1 , Mahmoud Ahmadian-A ttari 1 , and Reza Asv adi 2,* 1 F a culty of Electrical Engineering, K.N.T oosi Universit y of T ec h nology , T ehran, Iran. E-mail s: mahdinangir@ee.kn tu.ac.ir- mahmoud@eetd.kntu.ac.ir 2 F aculty of Electrical Engineering, Shahid Beheshti Univers ity , T ehran, Iran. E-mail: r asv adi@sbu.ac.ir * Corresponding Author: r asv adi@sbu.ac.ir Abstract: In this pap er, a practical co ding sc heme is designed for the binary W y ner-Z iv (WZ) problem b y using nested lo w-densit y generator-matrix (LDGM) and lo w-densit y parit y- c hec k (LDPC) co des. This sc heme con tains tw o steps in the enco ding pro cedure. The first step inv o lves applying the binary quan tization b y emplo ying LDGM co des and the second one is usin g the synd ro me-co ding tech nique b y utilizing LDPC co des. The deco ding algorithm of the prop osed sc heme is based on the Sum-Pro duct (SP) algorithm with the help o f a side information av ailable at the de co der side. It i s theoretically sho wn that the comp ound structure has the capabilit y of ac hieving the WZ b ound. The prop osed method approac hes this b o und b y utilizing the iterativ e message-passing algorithms in b oth enco ding and deco ding, although theoretical results show that it is asy mptotically ac hiev a ble. Keyw ords: W yner-Ziv problem, comp ound LDGM-LDPC codes, rate-distortion, message- passing algorithms, binary quan tization, syndrome-based deco ding. 1. Intro duction Recen tly , graph-based co des, i.e., co des with a graphical represen tation, are mostly utilized in c hannel coding, source co ding, and also in jo in t source-c hannel co ding sc hemes b ecause of their sup erior p erformance in ac hieving theoretical b ounds and lo w computational complex- it y . T urb o co des and LDPC co des with iterativ e deco ding algorithms are able to achiev e the capacit y of most of the kno wn comm unication channe ls [1, 2]. In addition, LDGM co des, whic h are the source co de dual of the LDPC co des, p erform v ery w ell in ac hieving the rate-distortion b ound of the binary-symmetric source [3], and the binary-erasure source [4]. Lossy source co ding with a side information av ailable at the deco der, kno wn as the W yner- Ziv (WZ) co ding, is a fundamen tal problem in distributed source co ding [5]. This problem also arises in practical applications, e.g., wireless sensor net w orks, and distributed video co ding. The WZ problem in the Gaussian domain has b een abundantly studied, ho w eve r , less atten t io n has b een paid to this problem in the binary con text. Previous studies ha ve used v arious types of source and chann el co ding sc hemes for dealing with the WZ problem. Briefly , if an efficien t com bination of a go o d source co de with a p erformance close to the Shannon rate-distortion limit and a go o d c hannel co de with a p erfor mance close to the c hannel capacity limit are utilized in this problem, then the WZ limit is ac hiev able [6, 7]. T o the b est of our kno wledge, there are few co de designs for the binary WZ problem. 1 Nested Con v olutional/T urb o co des are one of the most efficien t co des emplo y ed in the WZ problem, whic h ha v e b een prop osed in [7]. This w ork ha s a chie ved within 0 . 09 bits aw ay from the binary WZ limit for T urb o co de of length 3 × 10 5 . F urthermore, p olar co des a nd spatially-coupled LDPC co des hav e b een used for the Quadratic Gaussian WZ problem in [8], and for the binary WZ problem in [9], resp ectiv ely . Sartipi a nd F ekri ha ve presen ted a co ding sc heme for the binary WZ problem based on the parit y approach [10]. They hav e ac hiev ed 0 . 2 bits aw ay from the binary WZ limit for the LDPC co des of length ab out 1000. The aim of this pap er is designing an efficien t co de whic h approach es the binary WZ limit. Our prop osed sc heme is in the fra mew ork of comp ound LDGM-LDPC co des. LDGM and LDPC co des can b e join tly used to fo rm comp ound co des whic h b elong to the category of nested co des [11, 12]. Nested co des are applicable in most scenarios of co ding theory suc h as noiseless binning and m ulti- t erminal source co ding [12]-[13]. A crucial comp onent of eac h WZ co ding problem comprises a quan tization part. There exist some binary quan tization sc hemes suc h as Surv ey-Propagation and Bias-Propaga t io n (BiP) a lg orithms whic h efficien tly p erform close to the Shannon rate-distortion limit [14, 15]. In our prop osed metho d, whic h is based on the syndrome approach, the BiP algorithm is em- plo y ed a s a binary quan tizer. The prop osed construction uses optimized degree distribution of LDPC co des o v er t he Binary-Symmetric Channel (BSC). F urthermore, w e hav e designed LDGM co des based on the optimized LD PC co des, whose v ariable no de degrees follo w fro m P oisson distribution. The main con tribution of this pap er is designing a comp ound structure of LDGM-LD PC co des whic h p erforms muc h closer to the binary WZ limit as compared to the previous studies lik e [7], [9 ] and [10]. It has b een sho wn in [6] that comp o und LDPC-LDGM structures asymptotically ac hiev e the binary WZ limit. T his pap er attempts to design a practical co ding sc heme for the former information theoretical study b y emplo ying efficien t message- passing algo rithms. Performin g close to the theoretical limit in our prop osed sc heme stems from the low Bit Error Rate (BER) op eration of LDPC co des a nd the p erformance near the rate-distortion limit of the LDGM co de designs. W e hav e ac hiev ed ab out 0 . 0033 bits a wa y from the binary WZ limit for the compo und LDGM- LD PC co des of length ab out 10 5 when the correlation b et we en the source and side information is mo deled b y a BSC with the crosso v er probabilit y 0 . 25. The rest of this pap er is or g anized as follows . In Section 2, the problem definition and preliminaries of our prop osed sch eme, including the binary WZ problem, the comp ound LDGM-LDPC construction, and the syndrome-based deco ding sc heme are introduced. Next in Section 3 , enco ding and deco ding sc hemes using message-passing algorithms are describ ed. Then, the design o f low - densit y generator a nd parit y-c hec k matrices are presen ted by using linear a lgebra approac h and combin a torics. Our metho d of designing the nested low-den sity co des is also presen ted in Section 3. In Section 4, the sim ulation results and discu ssions ab out the rate-distortion p erformance of the prop osed co de design and its adv an tages are giv en. Finally , Section 5 dra ws the conclusion and future w ork. 2. Prelimina ries In this pap er, ve ctors and matrices are indicated by lo we r case a nd upp ercase b oldfa ced letters, respectiv ely . Scalars and realization of random v ariables are represe nted by lo we r case italic letters. 2 The co deb o o k of an individual co de is represen ted b y its corresp onding generator matrix. In T anner graph of co des, the v ariable and c heck no des are depicted b y circles and squares, respectiv ely . If all of the v ariable and c hec k no des hav e resp ectiv ely the same degrees d v and d c , then their asso ciated matrices are called ( d v , d c )-regular. F or irregular co des, t w o p olynomials denoted by λ ( x ) = P D v i =2 λ i x i − 1 and ρ ( x ) = P D c i =2 ρ i x i − 1 are used for determining their degree distributions fro m an edge p ersp ectiv e, where λ i and ρ i are fra ctions of edges connected to degree i v a riable and c hec k no des, resp ectiv ely [16]. The WZ problem is defined in [5]. Let s = ( s 1 , ..., s n ) b e a sequence of uniform binary input source; then we deal with a binary WZ problem. An enco ding function maps s to a lossy compressed v ector v with length k smaller than n . Moreov er, a deco ding function using v and side information j with length n deco des s to ˆ s . If the correlation b et we en s and j is mo deled b y a BSC with the crossov er probabilit y of p and distortion is ev aluated b y the Hamming distance measure, then the rate-distortion theoretical b ound f o r the binary WZ problem is as follows : R W Z ( D ) = l .c.e. { h ( D ∗ p ) − h ( D ) , ( p, 0) } , (1) for 0 ≤ D ≤ p . In (1), D ∗ p = D (1 − p ) + p (1 − D ) is the binary con v o lution, h ( x ) = − x log 2 x − (1 − x )log 2 (1 − x ) is the binary en trop y f unction, and l.c.e. stands for the low er con v ex en velop of the term h ( D ∗ p ) − h ( D ) and the p oin t ( p, 0) in the rate-distortion plane [5]. The total distortion in our design stems from b oth enco der and deco der. The enco der distortion is basically related to the mapping of s to the nearest (in the sense of Hamming distance) co dew ord x = ( x 1 , ..., x n ) fro m a co deb o ok, with a syndrome whic h is neces sarily zero; this mapping is called binary quan tization. In our design, x is selected from a comp ound LDGM-LDPC co deb o ok. At the deco der, distortion stems f ro m inefficiency whic h will b e negligible if an efficien t chann el error correcting co de is applied with a rate that is smaller than the capacit y of the correlation c hannel. Therefore, the tota l a v erage distortion is: D t = 1 n E [ n X i =1 d ( s i , ˆ s i )] = 1 n E [ n X i =1 d ( s i , x i )] ∗ 1 n E [ n X i =1 d ( x i , ˆ s i )] ∆ = d 1 ∗ d 2 , (2) where d ( ., . ) is the Hamming distortion measure, and E [ . ] denotes the exp ected v alue. 2.1. The Comp ound LDGM-L DPC Construction Comp ound construction consists of an m × n generator matrix G a nd a parit y-c heck matrix ˜ H ∆ = [ H T 1 , H T 2 ] T where [ . ] T stands for the matrix transp osition. T anner graph represen tation of a compo und construction is sho wn in Fig. 1. In this figure, filled circular no des represen t the source no des. Dimensions of matrices H 1 and H 2 are k 1 × m and k 2 × m , resp ectiv ely . These matrices result in t w o nested co des with generator matrices G and G 1 via the fo llo wing relations: G = ˜ G × G , G 1 = G 1 × G , (3) where ˜ G and G 1 are generator matrices suc h that ˜ G ˜ H T = 0 and G 1 H T 1 = 0 . The parit y- c hec k matrices of t he nested co des G and G 1 are denoted b y H and H 1 , resp ectiv ely . In 3 Fig. 1: T anner graph represen tation of a comp ound LDG M-LDPC co de construction our prop osed sc heme, w e hav e considered H to b e a parit y-c hec k matrix of an LDPC co de. F urthermore, some ro ws of H are ch o sen to form the submatrix H 1 whose generator matrix, G 1 , has v a r ia ble no de degrees whic h follow from a Pois son distribution [17]. Fig. 2: Blo c k Diagram of the Binary W yner-Ziv Co ding Sc heme The blo ck diagram of our co ding sc heme for the binary WZ problem is illustrated in Fig. 2. It consists of a binary quan tization using LDGM co des [18], whic h generates u = ( u 1 , ..., u m − k 1 ) and then y = ( y 1 , ..., y m ). Co dew ord x is giv en by: x = u × G 1 = u × G 1 × G = y × G . (4) The syndrome v ector of y is z 1 = y × H T 1 = 0 , a nd only syndrome z 2 = y × H T 2 is sen t to the deco der as a lo ssy compressed vec to r of s along with the side information j . In addition to these v ectors, the to t a l syndrome z is defined a s [ 0 , ..., 0 | {z } n − m , ( z 1 ) , ( z 2 )]. A t the deco der, a sequence ˆ s is deco ded utilizing the receiv ed syndrome z 2 and the side information j . This deco ding sc heme is used for the asymmetric Slepian-W olf structure [19]. Inspired by the syndrome - deco ding idea, if some syndrome bits are set to b e zero and only the remaining non-zero bits are sen t to the deco der, then a lossy co ding sc heme is emplo ye d for implemen ta tion of the binary WZ problem. 4 2.2. An Infor ma tion-Theo retic Viewp oint In what follo ws, the reason wh y our prop osed co ding sc heme p erforms close to the rate- distortion limit of the binary WZ problem is briefly explained. S upp ose that the source co ding r a te R s and the c hannel co ding rate R c are selected for the binary quantiz a tion and the syndrome-coding pro cedures , resp ectiv ely , such that the follow ing inequalities are satisfied for arbitrary small p ositiv e v alues of ε s and ε c , 1 − h ( d 1 ) ≤ R s < 1 − h ( d 1 ) + ε s , (5a) 1 − h ( d 1 ∗ p ) − ε c < R c ≤ 1 − h ( d 1 ∗ p ) . (5b) If a low BER c hannel deco ding algorithm is used in the syndrome-decoding step, then t he o ve rall distortion is D t = d 1 ∗ d 2 ≈ d 1 . In a comp ound construction, the tota l rate denoted b y R t equals the difference R s − R c [6], and hence: h ( D t ∗ p ) − h ( D t ) ≤ h ( d 1 ∗ p ) − h ( d 1 ) ≤ R t < h ( d 1 ∗ p ) − h ( d 1 ) + ε s + ε c | {z } ε ≈ h ( D t ∗ p ) − h ( D t ) + ε, (6) b ecause d 1 ≤ D t , and h ( x ∗ p ) − h ( x ) is a non-increasing function o f x . Therefore, if R s and R c are c hosen a ccording to (5), then R t b ecomes arbitrarily close to h ( D t ∗ p ) − h ( D t ), b ecause ε = ε s + ε c can b e arbitrarily small b y a prop er design of source and c hannel co des in the comp ound structure. Conseque ntly , this leads to achi eving the binary WZ b ound (1). 3. The Prop osed Scheme 3.1. The Enco ding Algor i thm The compression pro cedure of s consists of tw o steps: (Step 1) The source sequence s with length n is quan tized to a co dew ord x ∈ G 1 b y using the BiP algorithm. Let u denote the information bits after quan tization. Next, y = u × G 1 and x = y × G are calculated. (Step 2) The syndrome z 2 = y H T 2 is obtained, and then it is sent to the deco der. Th us, the ov erall compression rate in this sc heme is k 2 n . Consider the f ollo wing example for further clarification. F or simplicit y , short length vec to rs and small-size matrices are considered. Example 1: Supp ose that the source sequence is s = (1 , 0 , 0 , 1 , 1 , 0 , 0 , 1 , 0 , 0) with length n = 10. The follo wing matrices in the comp ound LDGM-LDPC construction a re also con- sidered with the size of m × n , k 1 × m , and k 2 × m , resp ectiv ely , for G , H 1 , and H 2 matrices, where m = 8, k 1 = 4, and k 2 = 2. G =               1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 1 1 0 0 1 0 0 0 1               , 5 ˜ H = H 1 H 2 ! =                    1 0 0 0 0 0 1 1 0 1 0 0 0 0 1 1 0 0 1 0 0 0 1 1 0 0 0 1 0 0 1 1          H 1 ( 0 0 0 0 1 0 1 1 0 0 0 0 0 1 1 1 ) H 2           . The generator matrices of LDPC and LDGM co des are calculated according to (3). These matrices are, respectiv ely , as follow s: G = 1 0 1 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 0 1 ! , G 1 =      0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 0 1 0 1 0 1      . F urthermore, one of the parit y-chec k matrices asso ciated with the LDPC co de G is giv en b y: H =               1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 0 1 1 0 0 0 0 0               . Using these matrices, the compressed sequence z 2 is obtained from the source sequen ce s . The LDGM co de G 1 has 2 ( m − k 1 ) = 16 co dew or ds with length n . The nearest co dew o r d to s is found by using a simple exhaustiv e search . Therefore, s is quan tized to t he co dew ord x = ( 1 , 0 , 1 , 0 , 1 , 0 , 0 , 1 , 1 , 0), whic h is the nearest co deword to s regarding the Hamming distance. The message sequence asso ciated with x is u = (0 , 0 , 1 , 0). By ha ving u , the lossy compresse d sequence z 2 is calculated, y = uG 1 = (1 , 1 , 1 , 1 , 0 , 0 , 1 , 0) , z 2 = y H T 2 = (1 , 1) . Finding the nearest co dew ord to s is done b y the BiP algorithm [15]. Supp ose that an LDGM co de with the asso ciated T anner graph is g iv en. Input bits of s a re lo cated in the source no des of its T anner g raph. In eac h round of the algorithm, bias v alues are calculated for the v ariable no des and they are compared with a threshold v alue of t , where 0 < t < 1. A ccording to this comparison, the v alues of some v ariable no des are fixed when their absolute bias v alues b ecome greater than t . If the bias v alue is p ositiv e, then the asso ciated v ariable bit will b e fixed to 0, a nd otherwise it will b e fixed to 1 . If there are no absolute bias v alues greater than t , then only a v ariable no de with the maxim um absolute bias v alue is fixed. This pro cess con tinues un til all the v ariable no des are fixed. Finally , u is obtained from the 6 v ariable no des. In the l -th iteration of each round, the message that is sen t from a ch ec k no de c a to a v ariable no de v i is: φ ( l ) c a → v i = Y v j ∈ ¯ N ( c a ) \{ v i } θ ( l ) v j → c a , (7) where ¯ N ( c a ) is the set of no des that are connected to the chec k no de c a , including the source no de s a , and θ ( l ) v j → c a is the message that has b een sen t from the v ariable no de v j to c a . Moreo v er, the message that is sen t from s a to c a is: θ ( l ) s a → c a = ( − 1) s a tanh( γ ) ∈ [ − 1 , 1] , (8) where γ is a real n umber that dep ends on the LDGM co de rate. Similarly , the message that is sen t f r o m v i to c a in the ( l + 1)-th iteration of eac h round is: θ ( l +1) v i → c a = Q c b ∈N ( v i ) \{ c a } (1 + φ ( l ) c b → v i ) − Q c b ∈N ( v i ) \{ c a } (1 − φ ( l ) c b → v i ) Q c b ∈N ( v i ) \{ c a } (1 + φ ( l ) c b → v i ) + Q c b ∈N ( v i ) \{ c a } (1 − φ ( l ) c b → v i ) , (9) where N ( v i ) is the set of c hec k no des that are connected to v i . The initial bias v alues θ (0) v i → c a are set to b e 1. Finally , the bias v alues θ v i for eac h v ariable no de are calculated at the end of eac h round after ˆ l iterations b y: θ v i = Q c b ∈N ( v i ) (1 + φ ( ˆ l ) c b → v i ) − Q c b ∈N ( v i ) (1 − φ ( ˆ l ) c b → v i ) Q c b ∈N ( v i ) (1 + φ ( ˆ l ) c b → v i ) + Q c b ∈N ( v i ) (1 − φ ( ˆ l ) c b → v i ) . (10) When the girth of the co de is 4, we can equip the BiP algorithm b y damping op eration to reduce the dep endency b et w een messages. Equation (9) will b e c hanged according to (3 . 10) and (3 . 11) in [15]. 3.2. The Deco ding Algo rithm The deco der receiv es side information j = s ⊕ ν a nd syndrome z 2 , whic h ⊕ sho ws binary addition and ν = ( ν 1 , ..., ν n ) is a ra ndom v ector, where ν i for 1 ≤ i ≤ n are Bernoulli i.i.d. random v ariables with parameter p . Then, the decoder finds the nearest sequence to j in the coset corresp onding to syndrome z = [ 0 , ..., 0 , ( z 2 )], using the Sum-Pro duct (SP) algorithm [20]. This algorithm is p erformed b y a n LD PC co de with the parit y-c heck matrix H . The follo wing example illustrates the decoding pro cedure of Example 1. Example 2: The syndrome z 2 = (1 , 1) is receiv ed, and the syndrome z 1 = y × H T 1 equals (0 , 0 , 0 , 0). The systematic form o f the parit y-c heck matrix of t he LDPC co de, whic h is used for designing the comp ound co de, is as follo ws: H sys =               1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 1 0 0               . 7 Therefore, z equals x H T sys = (0 , 0 , 0 , 0 , 0 , 0 , 1 , 1), whic h is the total syndrome that the de- co der receiv es. Besides, suppo se that the side information j = (1 , 0 , 1 , 1 , 1 , 0 , 0 , 1 , 0 , 1) is a v ailable at the deco der. The task of the deco der is finding the nearest sequence to the side information j , whic h has the total syndrome (0 , 0 , 0 , 0 , 0 , 0 , 1 , 1). There a re four sequences with length 10 whic h ha v e the same syndrome v alue z . They are: a = (0 , 0 , 0 , 0 , 0 , 0 , 1 , 1 , 0 , 0) , b = (0 , 1 , 1 , 0 , 0 , 1 , 0 , 1 , 0 , 1) , c = (1 , 0 , 1 , 0 , 1 , 0 , 0 , 1 , 1 , 0) , d = (1 , 1 , 0 , 0 , 1 , 1 , 1 , 1 , 1 , 1) . The nearest sequence to the side information j is c , so ˆ s is equal to c . Note that ˆ s = x declares that there is no distortion in the deco ding part. 3.3. Co de Design In this subsection, design pro cedure of lo w-densit y graph-based co des in the comp o und construction is described. In this regard, some definitions and lemmas are provide d. Definition 1: The d i a gonal elemen ts of an m × n matrix ( m ≤ n ) consist of its ( i, i )-th en tries, for i = 1 , 2 , ..., m . L emm a 1: The degree distribution of a binary matrix A remains the same under an y column and/or row in terchange. Definition 2: An al l- one-diagonal binary matrix is a matrix in whic h the main d i ago- nal entrie s are one a nd the other en tries, i.e., the off-diagonal en tries, are arbitrary bits. Similarly , an al l-zer o-diagonal binary matrix is a matrix with zero-v alued main diagonal en tries. L emm a 2: F or any binary full-rank matrix A , there is at least one p erm utation of columns and/or ro ws, suc h that the resulting matrix is an al l-one-diagon al matrix. Pr o of: See App endix A. L emm a 3: Let I b e an m × n binary al l-one-diago nal matrix, and all of the off-diagonal en tries are zero, i.e, rectangular iden tit y matrix. Let a lso A b e an m × n al l-one-diagon a l , binary full-rank matrix with degree distribution ( λ ( x ) , ρ ( x )). Suppose that A 0 obtains from A by inserting zero-v alued entrie s in the main d i a gonal . Then the following 2 m × 2 n matrix H has the same degree distribution ( λ ( x ) , ρ ( x )), H = I A 0 A 0 I ! . Pr o of: See App endix B. In our prop osed sc heme, an ( n − m + k 1 + k 2 ) 2 × n 2 full-rank parity-c heck matrix is selected for a specified degree distribution ( λ ( x ) , ρ ( x )). T hen, it is transformed in to an al l - o ne- diagonal matrix A with the same degree distribution by using prop er row and/or column p erm utations. A ccording to Lemma 1 and Lemma 2, there exists at least one matrix with these prop erties. The ( n − m + k 1 + k 2 ) × n parity-c hec k matrix H of an LDPC co de in the comp ound structure is formed with the same degree distribution ( λ ( x ) , ρ ( x )) by using Lemma 3. If I 8 is an ( n − m + k 1 + k 2 ) 2 × n 2 rectangular iden tit y matrix, then the first n − m + k 1 ro ws of H are c hosen, in order to obtain a submatrix H 1 as a parit y-che ck matrix of the LDGM co de G 1 . Supp ose the remaining k 2 ro ws are placed in another submatrix H 2 . If ( n − m + k 1 ) ≤ k 2 , then H 1 is an al l-one-di a g onal matrix. Since, there are sev eral generator matrices fo r a giv en parit y-c heck matrix H 1 , obtaining an ( m − k 1 ) × n sparse g enerator matrix G 1 is desired suc h that the we ight of its row s follo ws a Pois son distribution function. Ob viously , t he parity - c hec k matrix H 1 is in the form of  I O B  , where I and O are respectiv ely an ( n − m + k 1 ) × ( n − m + k 1 ) iden tity matrix, and a n ( n − m + k 1 ) × ( m − k 1 − n 2 ) all-zero matrix. In addition, B consists of the first n − m + k 1 ro ws of A 0 and it is a n ( n − m + k 1 ) × n 2 matrix. F or designing G 1 , it is necessary and sufficien t to find m − k 1 linearly indep enden t co de- w ords of it. Assume n -tuple c j is a co dew ord of G 1 , for j = 1 , 2 , ..., m − k 1 , i.e., H 1 c T j = 0 T . Split the co dew ord c j in to tw o message parts m j, 1 ∈ { 0 , 1 } m − k 1 − n 2 and m j, 2 ∈ { 0 , 1 } n 2 , and a p arity part p j ∈ { 0 , 1 } n − m + k 1 , suc h that c j = ( p j , m j, 1 , m j, 2 ). T o design the ro ws of G 1 , m j ∆ = ( m j, 1 , m j, 2 ) is filled with m − k 1 differen t v ector of information bits, whose Hamming w eight is w H ( m j ) = ζ , where w H ( b ) denotes the Hamming weigh t of the ve ctor b . Then, the p a rity part p j satisfies I p T j =  O B  m T j = B m T j, 2 . Therefore, p j equals binary sum of ζ columns of the submatrix  O B  . Suppose these columns, denoted b y b q j,i for j = 1 , 2 , ..., m − k 1 , i = 1 , 2 , ..., ζ , and q j,i ∈ { 1 , 2 , ..., m − k 1 } a re sorted suc h that w H ( ⊕ ζ i =1 b q j,i ) ≤ w H ( ⊕ ζ i =1 b q j +1 ,i ) for j = 1 , 2 , ..., m − k 1 − 1, where ⊕ ζ i =1 sho ws the binary sum ov er index i . Let p ( i ) = e − λ λ i i ! b e the probabilit y mass function of a P o isson distribution with parameter λ , for i = 0 , 1 , 2 , ... . Also consider { a j } m − k 1 j =1 is a monotonically increas- ing sequenc e of ordered in teger n um b ers, with a j ∈ { 1 , 2 , ..., i max } fo r a fixed inte ger n um b er i max . In this sequence, the probabilit y of o ccurren ce of an y in teger n umber i ∈ { 1 , 2 , ..., i max } equals p ( i ), or equiv alen t ly , the num b er of o ccurrences of i is n i = [ p ( i ) × ( m − k 1 )], where [ x ] sho ws the nearest in teger num b er to x . Obvious ly , P i max i =1 n i ≈ m − k 1 . By these assumptions , the Hamming w eigh t of m j, 1 is set as follo ws: w H ( m j, 1 ) = j a j − ( w H ( ⊕ ζ i =1 b q j,i ) + ζ ) k + , (11) where ⌊ x ⌋ + is x for x ≥ 0, and otherwise is zero. Then, t he p ositions of 1’s in m j, 1 are c hosen in a wa y that the resulting v ectors b e linearly indep enden t. Note that the Ha mming w eight of m j, 1 do es not a ffect p j . Hence, m j, 1 is filled with some information bits whose we ig hts are ch o sen according to (11). By this pro cedure, the row s of G 1 are linearly indep enden t co dew o r ds c j , whose Hamming w eigh ts satisfy the follow ing inequalit y , for j = 1 , 2 , ..., m − k 1 . w H ( c j ) = w H ( p j ) + w H ( m j, 1 ) + w H ( m j, 2 ) = w H ( ⊕ ζ i =1 b q j,i ) + j a j − ( w H ( ⊕ ζ i =1 b q j,i ) + ζ ) k + + ζ . (12) If a j ≥ w H ( ⊕ ζ i =1 b q j,i ) + ζ , then w H ( c j ) ≤ a j . Otherwise, if a j < w H ( ⊕ ζ i =1 b q j,i ) + ζ , then w H ( c j ) ≤ ζ (max l { w H ( b l ) } + 1). In the latter case, it is sufficien t to tak e the in teger n um b er i max suc h that i max ≥ ζ (max l { w H ( b l ) } + 1). Therefore, in b oth cases w H ( c j ) ≤ i max . Since these co dew ords are sparse v ectors, G 1 will b e an LD GM co de with the Poiss o n degree 9 distribution for v ariable no des. There are some limitations rega rding the blo c k lengths that should b e considered in design and implemen tation, they are men tioned in the follo wing lemmas. L emm a 4: In our co de design, the followin g inequalitie s are satisfied in the comp ound structure, n − k 2 ≤ m − k 1 , n 2 ≤ m − k 1 . (13) Pr o of: See App endix C. L emm a 5: In the comp ound structure, if k 1 + k 2 ≤ m ≤ 2 k 1 + k 2 is satisfied fo r the giv en nested co des G 1 , G , and the parit y-c heck matrix ˜ H , then the existence of at least one binary matrix G is guaran teed. Pr o of: See App endix D. The aforemen tioned inequalities are actually practical limitations in the selection of blo c k lengths. Ho we ve r, there exist tw o imp ortan t information theoretic limitations in our design. Firstly , if the compression rate and t he resulting distortion of Step 1 a r e, resp ectiv ely , consid- ered to b e R 1 and d 1 , then R 1 > 1 − h ( d 1 ) should b e satisfied. Secondly , the rate R 2 should b e smaller than the capacit y o f correlation chann el b et w een the quantiz ed sequence x and the side informatio n j in the syndrome-co ding. In other w ords, if R 2 < 1 − h ( p ∗ d 1 ), then the deco ding distortion d 2 or its asso ciated BER in the syndrome-deco ding step can b e arbi- trarily small. In fact, the main result of our practical co ding sche me is based on exploiting these limitations b y an efficien t co de design whic h is stated in the follow ing theorem. The or em 1: The comp ound LDGM-LDPC structure is able to ach iev e the binary WZ limit if it satisfies b oth of the fo llo wing features: 1. An efficien t lossy source co ding algorithm is used for the binary quan tization by whic h the rate-distortion limit is ac hiev able. 2. A low BER c hannel deco ding algorithm is applied in the synd ro me-decoding step. Pr o of: See App endix E. The BiP and the SP a lg o rithms are a ble to satisfy b oth of these conditions in the Theorem 1. How eve r, iterativ e message-passing algorithms are sub-optimal and there is a sligh t ga p b et w een theoretical limits and the ra t e- distortion p erformance of these algorithms. 4. Numerical Results and Discussion In this section, some sim ulatio n results are presen ted to demonstrate the p erformance of the prop osed co ding sc heme at differen t ra tes. Optimiz ed degree distributions of irregular LDPC co des ov er the BSC are used in our sim ulations. A dditionally , the prop osed sc heme is also implemen ted by using regular parity-c hec k matrices. The degree distributions of LDPC co des ha v e b een presen ted in App endix F 1 . Our results are exhibite d for tw o cases of correlation b etw een s and j where the parameter p equals 0 . 25 and 0 . 05. F urthermore, an example of the comp ound co des is provi ded for whic h their parameters are adjusted to those used in [9] and [10]. 1 The degree distributions are obtained f rom [21]. The matrices are generated according to the degree distributions b y us ing the progressi v e edge-gro wth (PEG) algorithm [22]. 10 The rate-distortion curv es of differen t co ding sc hemes ar e presen ted in Figs. 3 and 4 with the same co de length and the same correlation parameter. It is apparen t that the rate-distortion p erformance of the prop osed metho d p erforms b etter than other tec hniques applied in [9] and [10]. This adv a n tage b ecomes greater a s the parameter ζ increases as far as the co des remain sparse. F or instance, rate-distortion p erformance o f the prop o sed sc heme is depicted for ζ = 1 , 2 , 5 and 10. In implemen tat io n of the message-passing algo r ithms, time sharing b et w een the p o in ts of ( p, 0) and ( D b + ε 0 , R b ) is emplo ye d where ( D b + ε 0 , R b ) is an achie ved rate-distortion p oin t, and ( D b , R b ) denotes the b oundary p oin t on the binary WZ limit curv e. The b o undary p oin ts are calculated (0 . 088 , 0 . 444) and ( 0 . 0 0 14 , 0 . 2764) for the correlation parameter v alues of 0 . 25 a nd 0 . 05, resp ectiv ely . These p oin ts separate the WZ limit curv e in to the high-rate and lo w-rate regions. The high-rate region is in the form of h ( p ∗ D t ) − h ( D t ) for distortion D t , and the lo w-rate region is a linear curv e. The blo c k size o f matrices, the length of co des, co de rates, and distortion v alues are presen ted in T ables 1 and 3 for irregular co des with ζ = 10 and in T ables 2 and 4 for regular co des. In these tables, d 1 and R 1 indicate distortion and rate of the binary quan tization step, resp ectiv ely . Similarly , d 2 and R 2 denote distortion a nd rate of LD PC co des used in the syndrome-decoding step, resp ectiv ely . W e also use D t and R t to denote total distortion and r a te v a lues, resp ectiv ely . F or more in tuition, let D w z b e the distortion v alue of the binary WZ limit at the rate R t . The rates R 1 , R 2 , and R t are calculated as follow s: R 1 = m − k 1 n , R 2 = m − k 1 − k 2 n , R t = k 2 n = R 1 − R 2 . (14) The distortion v alues d 1 , d 2 , and D t are obtained fro m the calculation of the av erage Ham- ming distance b etw een the source and the deco ded sequ ences a ccording to (2). In fact, tw o t yp es of gaps can b e defined to compare the results from rate and distortion p oin t of view. The difference b etw een an achie ved distortion (rate) and its asso ciated theoretical limit is called the gap of distortion (gap o f rate). The amoun t of D t − D w z determines the resulting distortion gap v alue for a giv en R t that is used in the ta bles. The gap of rate is appro xi- mately prop ortional to the ga p of distortion with the ra t io R b / ( p − D b ) due to the g eometric similarit y . The parameters of λ and i max emplo y ed in designing the irregular co des are men tioned in T ables 1 and 3. Eac h p oint in our sim ulations has applied 100 randomly generated source sequenc es s with uniform distribution. The v alue o f maxim um iteration ˆ l is set to b e 25 in eac h round of the BiP algorithm. Moreo v er, w e hav e set t = 0 . 8 and γ ≈ 2 R 1 = 2 m − k 1 n in our sim ulatio ns. In addition, the maxim um n umber o f iterations in t he SP algorithm is set to b e 100. Example 4: Supp ose the correlation parameter p equals 0 . 25. The blo ck lengths, ra t es, and distortion v alues of the comp o und sch eme are presen ted for irregular and regular co des, respectiv ely , in T a ble 1 and T able 2. The rate-distortion p erformance of our prop osed sc heme is also illustrated in Fig . 3 for this correlation para meter. F or instance, in the case of R t equals 0 . 6, the achie ved distortion gap for the prop osed sch eme is 0 . 0033 bits aw ay from the WZ limit. Ho w ev er, the gap v alues at the same rate are 0 . 079 and 0 . 066 bits p er channe l use for co ding sc hemes [10] and [9], resp ectiv ely . The equiv alen t gaps in the sense of distortion are, respectiv ely , 0 . 0288 and 0 . 0 2 4 bits. It is apparen t that the distortion gap v a lues for the co ding metho ds in [10] and [9] are ab out eigh t t imes more than that of the prop osed sc heme 11 due to inheren t defects of their metho ds. The co ding sc heme of [9] has imp erfections whic h lead to a considerable a moun t of ga p, suc h as: only r egular LDGM and LDPC co des are used in that structure whic h migh t b e substituted b y irregular co des. Moreo ver, there is a p ossibilit y of failure in the algorithm o f [9] that causes rep etition of the encoding pro cess . Hence, the p erfo r mance of the sche me is not suitable for short length co des as a result of the attempt at enco ding source sequenc es b y suc h a rep etitiv e pro cess. In the prop osed sc heme of [10] that is based on sending parit y bits, using a systematic c hannel encoder is essen tial. F or this reason, a Gaussian elimination should b e applied b efore enco ding that increases the complexit y of enco ding. The designed LDPC co des in [10] are based on the MacKa y co des whic h are not optimal co des. Alternativ ely , degree distribution of the co des can b e replaced with optimized co des based on sending either parit y bits or some parts of it. T able 1 Sim ula tion Results for p = 0 . 25 and ζ = 1 0- Irregular Co des (Example 4) Code n m k 1 m − k 1 k 2 ( d 1 , R 1 ) ( d 2 , R 2 ) R t D t D wz λ i max 1 100000 76800 20000 56800 44400 (0 . 0892 , 0 . 568) (0 . 0037 , 0 . 124) 0 . 444 0 . 0922 0 . 088 873 . 2 2000 2 100000 9 0000 263 00 63700 50000 (0 . 0695 , 0 . 637) (0 . 0036 , 0 . 137) 0 . 5 0 . 0726 0 . 0688 855 . 5 2000 3 100000 9 5700 200 00 75700 60000 (0 . 0403 , 0 . 757) (0 . 003 , 0 . 157) 0 . 6 0 . 0431 0 . 0398 714 . 95 1600 4 100000 10720 0 20000 87200 70000 (0 . 0181 , 0 . 872) (0 . 003 , 0 . 172) 0 . 7 0 . 021 0 . 017 694 . 0 5 1 6 00 5 100000 12000 0 21800 98200 80000 (0 . 0021 , 0 . 982) (0 . 003 , 0 . 182) 0 . 8 0 . 0051 0 . 0011 707 . 25 1600 T able 2 Sim ulation Results for p = 0 . 25 - Regula r Co des (Ex ample 4) Co de n m k 1 m − k 1 k 2 ( d 1 , R 1 ) ( d 2 , R 2 ) R t D t D wz 6 100000 70000 13600 56400 4440 0 (0 . 097 , 0 . 564) (0 . 025 , 0 . 12) 0 . 444 0 . 1172 0 . 088 7 100000 90000 2 8000 6 2 000 50000 (0 . 0793 , 0 . 62) (0 . 0235 , 0 . 12) 0 . 5 0 . 0991 0 . 0688 8 100000 95000 2 0000 7 5 000 60000 (0 . 049 , 0 . 75) (0 . 0244 , 0 . 15) 0 . 6 0 . 071 0 . 0 3 98 9 100000 105000 20000 85 0 00 70000 (0 . 0283 , 0 . 85) (0 . 0221 , 0 . 15) 0 . 7 0 . 0491 0 . 017 10 100000 120000 25000 95000 8000 0 (0 . 0133 , 0 . 95) (0 . 024 , 0 . 15) 0 . 8 0 . 0367 0 . 0011 Example 5: In this example, the correlation parameter p is assumed to b e 0 . 05. The blo c k lengths, rates, and distortion v alues of the comp ound sc heme are presen ted f o r irregular co des with ζ = 10 and for regular co des, resp ectiv ely , in T able 3 a nd T able 4. The rate-distortion p erformance of our prop osed sc heme is also depicted in Fig. 4 for the same correlation parameter. In the b oundary p oin t, R b = 0 . 2764 , we a c hiev e 0 . 0016 bits a wa y from the WZ limit, ho w eve r the gap v alues o f rate at the same p oin t are 0 . 0396 and 0 . 0347 bits p er c hannel use for the metho ds in [10] and [9], respectiv ely . The equiv alen t gaps of distortion are, respectiv ely , 0 . 007 a nd 0 . 0061 bits for the metho ds in [1 0] and [9] whic h are ab out four times more than that of the prop o sed metho d. T able 3 Sim ula tion Results for p = 0 . 05 and ζ = 1 0- Irregular Co des (Example 5) Code n m k 1 m − k 1 k 2 ( d 1 , R 1 ) ( d 2 , R 2 ) R t D t D wz λ i max 11 100000 18 0 000 81760 98240 27640 (0 . 0 0 17 , 0 . 98 24) (0 . 0013 , 0 . 706) 0 . 2764 0 . 003 0 . 0014 442 . 7 1000 12 Fig. 3: Rate-Distortion p erformance of differen t co ding sc hemes for p = 0 . 25 (Example 4) T able 4 Sim ulation Results for p = 0 . 05 - Regula r Co des (Ex ample 5) Co de n m k 1 m − k 1 k 2 ( d 1 , R 1 ) ( d 2 , R 2 ) R t D t D wz 12 100000 180000 82360 97640 2764 0 (0 . 0035 , 0 . 9764) (0 . 0 154 , 0 . 7) 0 . 2 7 64 0 . 0188 0 . 0014 It is a pparent from Figs. 3 and 4 that the rate-distortion p erformance of our sc heme is v ery close to the binary WZ limit. It is p ossible that w e can ac hiev e a closer rate-distortion p erformance to the WZ limit for small rates b y using time sharing. Nev ertheless, the gap v alue can b e further reduced b y increasing the blo c k length and the parameter ζ . Example 6: This example presen ts sim ulation results o f applying the prop osed sch eme b y emplo ying the same parameters used in [10] and [9]. Th us, the co de lengths and the correlation parameters, i.e., p = 0 . 27 , 0 . 134, are c hosen according to them. The utilized parameters and the results are presen ted fo r the irregular co des with ζ = 10 in the first and second ro w of T able 5 asso ciated to [10] and [9], resp ectiv ely . The achie ved distortion gap b y the prop osed algorithm are also depicted at the last column of T able 5 for more comparison. In [10], a gap of ra te 0 . 155 bits p er ch a nnel use aw ay from the WZ limit has b een achiev ed when the co de length equals 3000. This is equiv alent to 0 . 058 1 bits of distortion gap. Instead, the distortion g ap of the prop osed metho d is 0 . 0387 bits aw ay from the binary WZ limit, where it is ab out one and a half times less than the one in [1 0]. Similarly , in [9], the a chiev ed gap of rate is 0 . 02 22 bits p er c hannel use for the co de length 1 40000. In our sc heme, the gap of distortion is 0 . 0037 bits for the same parameters. It is equiv alent to 0 . 0151 bits p er c hannel use of gap in rate. By using time sharing in the linear part of the curv e, the gap of rate decreases to 0 . 0029 bits p er c hannel use, where it is ab out eigh t times less than the one in [9]. 13 Fig. 4: Rate-Distortion p erformance of differen t co ding sc hemes for p = 0 . 05 (Example 5) T able 5 P a rameters a nd Res ults of our scheme in a reverse comparison- ζ = 10 (Ex ample 6) Code n m k 1 m − k 1 k 2 ( d 1 , R 1 ) ( d 2 , R 2 ) R t D t D wz λ i max Gap 13 3000 2 000 548 1452 1212 (0 . 1 3 8 , 0 . 484) (0 . 0148 , 0 . 08) 0 . 40 4 0 . 1487 0 . 11 128 . 77 300 0 . 0387 14 14000 0 18000 0 57400 1 22600 67300 (0 . 0173 , 0 . 8757) (0 . 0025 , 0 . 395) 0 . 4807 0 . 0197 0 . 016 836 . 92 2000 0 . 0037 It is notew orthy that there exists at least o ne binary matrix G in the compound LDGM- LDPC structure, b ecause the condition k 1 + k 2 < m < 2 k 1 + k 2 is satisfied for all co des of the T ables according to Lemma 5. 5. CONCLUSION In this pap er, we prop osed an efficien t co ding sc heme f o r achie ving the binary WZ limit em- plo ying a comp ound LDGM-LDPC co de construction, whic h fulfills a nested co ding struc- ture. W e utilized optimized degree distribution of LDPC co des and the asso ciated LD GM co des nested with the optimized co des. F urthermore, v aria ble no de degrees of LDGM co des are designed to b e a P o isson distribution. In our sc heme, if efficien t source and ch a nnel co des are used with the capabilit y o f ac hieving the rate-distortion and the capacit y limits, the p erformance of the comp o und co de gets closer to the binary WZ limit with a ny arbi- trary precision. W e applied the BiP a lgorithm for the binary quan tization using LDGM co des and the SP algo rithm for the syndrome-dec o ding using LDPC co des. By emplo ying these algorithms, our sim ulation resul t s confirmed that the rate-distortion p erformance of the prop osed sc heme is v ery close to the binary WZ theoretical limit. F uture study may extend the prop osed co ding sc heme and the iterativ e message-passing algorithms for multi -terminal source co ding scenarios. Designing m ulti-terminal quan tization 14 and joint deco ding algorithms is considered in o ur future researc h study . Moreo ver, using impro v ed message-passing a lgorithms for reach ing smaller gap from the theoretical limit remains a future researc h t o pic. 6. References [1] Kbaier Ben Ismail, D., D o uillard, C., Kerouedan, S.: ’Impro ving irregular turb o co des’, IET Electronics Letters., 2011, 47 , (21), pp. 1184- 1186 [2] Hareedy , A. H., Khairy , M. M.: ’Selectiv e max-min algorithm for lo w-densit y parity- c hec k deco ding’, IET Comm unications, 2013, 7 , (1), pp. 65-70 [3] Sun, Z., Shao, M., Chen, J., W ong, K.M., W u, X.: ’A c hieving the ra te-distortion b ound with lo w-densit y generator matrix co des’, IEEE T rans. Comm un., 201 0, 58 , (6), pp. 1643-1653 [4] Martinian, M., Y edidia, J.: ’Iterativ e quan tization using co des on graphs’ . 41th Allerton Conf. on Comm unication, Con trol, and Computing, Allerton House, Montice llo , IL, USA, Oct 2003 [5] W yner, A., Ziv, J.: ’The rate-distortion function for source co ding with side information at the deco der’, IEEE T ra ns. Inf. Theory , 197 6 , 22 , (1), pp. 1- 10 [6] W ain wrigh t, M.J., Martinian, M.: ’Low - D ensit y Graph Co des That Are Optimal for Binning and Co ding With Side Information’, IEEE T rans. Inf. Theory , 2009, 55 , (3), pp. 1061-107 9 [7] Liv eris, A.D ., Xiong, Z., Georghiades, C.N.: ’Nested con v olutional/turb o co des for the binary W yner-Ziv problem’ . In ternational Conference on Image Pro cessing, Barcelona, Spain, Sept 2003, pp. 601-4 [8] Eghb a lian-Arani, S., Behro ozi, H.: ’Polar Co des for a Quadratic-Ga ussian W yner-Ziv Problem’ . Pro ceedings of 10 t h ISW CS 2013, Ilmenau, German y , A ug 20 13, pp. 1- 5 [9] Kumar, S., V em, A., Nara y a nan, K., Pfister, H.D.: ’Spatially - coupled co des for side- information problems’ . IEEE ISIT, Honolulu, HI, USA, June 20 1 4, pp. 516-520 [10] Sartipi, M., F ekri, F.: ’Lo ssy distributed source co ding using LDPC co des’, IEEE Com- m un. L ett., 2009, 13 , (2), pp. 13 6-138 [11] Hsu, C. H.: ’Design and Analysis of Capacit y-ac hieving Co des and Optimal Receiv ers with Lo w Complexit y’ . PhD thesis, D ept. Elect. Eng., Univ. Mic higan, Ann Arb or, 2006 [12] Kelley , C. A., Kliew er, J.: ’Algebraic constructions of graph-based nested co des from protographs’ . IEEE ISIT, Au stin, TX, USA, July 2010, pp. 829-833 [13] Liu, L., Ling, C.: ’P olar lattices are g o o d for lossy compression’, IEEE Infor matio n Theory W orkshop - F all (ITW), Jeju, South Korea, Oct 2015 , pp. 342- 346 [14] W ain wrigh t, M. J., Manev a, E., Martinian, E.: ’Lossy Source Compression Using Lo w- Densit y Generator Matrix Co des: Analysis and Algorithms’, IEEE T rans. Inf. Theory , 2010, 56 , (3), pp. 1351- 1 368 15 [15] Filler, T.: ’Minimizing em b edding impact in steganograph y using lo w densit y co des’ . Master’s thesis, SUNY Bingham ton, 2007 [16] Richards o n, T., Urbank e, R.: ’Mo dern co ding theory’ (Cam bridge Unive rsity Press, 2008) [17] Aref, V., Macris, N., V uffray , M.: ’Approac hing the Rate-D istortion Limit With Spatial Coupling, Belief Propagation, and Decimation’, IEEE T rans. Inf. Theory , 2015, 61 , (7), pp. 3954-397 9 [18] Filler, T., F ridrich, J.: ’Binary quan tization using Belief Propagat io n with decimation o ve r factor graphs of LD GM co des’ . 45th Allerton Conf. o n Comm unication, Control, and Computing, Allerton House, Mon ticello, IL, USA, Sept 2007 [19] Pradhan, S. S., Ramchandran, K.: ’Distributed source co ding using syndromes (DIS- CUS): design and construction’, IEEE T ra ns. Inf. Theory , 2003, 49 , (3), pp. 626- 643 [20] Live ris, A. D., Xiong, Z., Georghiades, C. N.: ’Compression of binary sources with side information at the deco der using LDPC co des’, IEEE Comm un. Lett. , 2002, 6 , (10), pp. 440-442 [21] Sch o n b erg, D. H.: ’Practical distributed source co ding and its application to the com- pression of encrypted data’ . PhD thesis, Univ. California, Berke ley , 20 0 7 [22] W ang, P ., Jin, Q., Xu, S., Y ang, H., Rash v and, H.F.: ’Efficie nt construction of irregular co des with midterm blo c k length and near-shannon p erformance’, IET Comm unications, 2011, 5 , (2), pp. 222-2 30 7. App endices 7.1. App endix A: Pro of of Lemma 2 W e pro v e the prop osition of this lemma by using the mathematical induction o n m . Supp ose that the en tries of matrix A =  A j,i  are denoted by A j,i , for 1 ≤ j ≤ m and 1 ≤ i ≤ n . This elemen t is lo cated on the j -th ro w and the i -th column. Basis for the induction: F or m = 1 , A b ecomes an 1 × n matrix. Since A is full-rank, so it has at least one en try that is 1. Clearly , it is mov ed to the first column of A to result in an al l- o ne-diagonal matrix. Induction h yp othesis: Supp ose that for any binary full-rank matrix A with m − 1 rows , there is a t least o ne p erm utation of columns and/or row s, suc h that the resulting matrix is an al l-on e-diagonal matrix, ( m > 1). Induction step: Now, w e w an t to sho w that the lemma is true f or a n y m × n binary full-rank matrix A = ( a 1 , a 2 , ..., a n ), where a i is the i -th column of A . Consider a column a i , i ∈ { 1 , 2 , ..., n } , whose Hamming weigh t is w , i.e., there are w non-zero entrie s in this column. Supp o se that these w non-zero en tries are placed in the rows { r α 1 , r α 2 , ..., r α w } , and the remaining ro ws of column a i are in the set { r 1 , r 2 , ..., r m }\{ r α 1 , r α 2 , ..., r α w } , where { r 1 , r 2 , ..., r m } is the set of ro ws of A , a nd \ denotes the set-reduction. If w = 1, we remov e the row r α 1 and the column a i from the matrix A , the resulting ( m − 1) × ( n − 1) matrix, called B , will b e full-rank, b ecause all of the elemen ts of the column 16 a i except for the row r α 1 are zero. A ccording to the induction h yp othesis, w e can tr a nsform B to an a l l-on e-diagonal matrix, b y a prop er p erm utation of columns a nd/or ro ws. No w, it is sufficien t to lo cate r α 1 to the m -th ro w, and a i to the m -th column. Let r α 1 → r m and a i ↓ a m sho w lo cating the row r α 1 in the row m , and lo cating the column a i in the column m , resp ectiv ely . If w > 1, w e claim that there is a t least one row in { r α 1 , r α 2 , ..., r α w } , suc h that after remo ving it and the column a i , the resulting ( m − 1) × ( n − 1) matrix is full-rank. Otherwise, for an y row r α ∈ { r α 1 , r α 2 , ..., r α w } , after removin g r α and the column a i , the remaining matrix, called B α , will not b e a full-rank matrix. Th us, there is a subset of ro ws o f B α , call this subset S α , whose sum of elemen ts is an all-zero v ector. Obv iously , r α / ∈ S α for all α ∈ { α 1 , α 2 , ..., α w } , and S α ⊆ { r 1 , r 2 , ..., r m }\{ r α } . F or an y α ∈ { α 1 , α 2 , ..., α w } , S α con tains an o dd n umber of rows in { r α 1 , r α 2 , ..., r α w } ; unless otherwise, if S α con tains an ev en num b er of ro ws in { r α 1 , r α 2 , ..., r α w } , then the sum of its elemen ts will b e all-zero, including the elemen t in the column a i . This con tradicts with the full-rank a ssumption of A . Hence, S α 1 con tains an o dd nu mber of ro ws in { r α 1 , r α 2 , ..., r α w } ; e.g., supp ose t hese rows are { r α ℓ , ... } , where ℓ 6 = 1. Similarly , S α ℓ con tains an o dd n umber of rows in { r α 1 , r α 2 , ..., r α w } , and r α ℓ / ∈ S α ℓ . No w, the sum of elemen ts of S α 1 and S α ℓ leads to an all-zero v ector, including the elemen t in the column a i . This also con tradicts with the full-rank assumption of A . Clearly , S α 1 6 = S α ℓ , b ecause r α ℓ ∈ S α 1 , but r α ℓ / ∈ S α ℓ . Therefore, there exists at least one elemen t in the column a i , that is 1 (consider it is in the row r j , i.e., A j,i = 1); suc h that af ter remo ving the row r j and the column a i , the resulting ( m − 1) × ( n − 1) matrix is full-rank. No w, it is sufficien t to a pply the induction h yp othesis and lo cate r j to the m -th ro w, and a i to the m -th column, i.e., r j → r m and a i ↓ a m . As a result, an al l-one-di a gonal matrix is built with the dimension m × n . 7.2. App endix B: Pro of of Lemma 3 The dimension of matrix H is twice the dimension of matrix A . Also, the n umber of columns or ro ws with a sp ecific weigh t in the matrix H are doubled in comparison with those of the matrix A . Therefore, the degree distribution remains unc hanged. 7.3. App endix C: Pro of of L emma 4 The first inequalit y (13) is true, b ecause n − m + k 1 ≤ k 2 , and the second inequalit y is true due to the dimension of matrix O . 7.4. App endix D: Pro o f of Lemma 5 The inequalit y k 1 + k 2 ≤ m is trivial and it is alw ays satisfied due to the dimension of matrix ˜ H . W e demonstrate that if m ≤ 2 k 1 + k 2 , then there exists at least o ne ch o ice for G with mn unknow n elemen ts. The tota l nu mber of know n elemen ts is n ( m − k 1 ) + n ( m − k 1 − k 2 ), whic h are o btained from (3). These equations are linearly indep enden t, and they are consisten t with eac h o ther due t o the nesting prop ert y of the comp ound co des. Note that in equations (3), G 1 and ˜ G are know n, b ecause ˜ H is giv en. Therefore, if m ≤ 2 k 1 + k 2 then n ( m − k 1 ) + n ( m − k 1 − k 2 ) ≤ mn , and hence it implies that there exists at least one c hoice fo r G suc h that (3) is satisfied f or the give n G 1 , G , and ˜ H . Therefore, the pro of is completed. 17 7.5. App endix E: Pro o f of Theo rem 1 Condition 1 results in (5a ) for a small v alue of ε s . Similarly , condition 2 means that (5b) yields d 2 ≈ 0 for a small v alue of ε c . There f ore, the total rate R t = R s − R c in the comp ound LDGM-LDPC structure will b e close to the binary WZ theoretical b o und as in (6). 7.6. App endix F: Degree Distribution of LDPC Co des The follo wing degree distributions of irregular co des are used in T ables 1, 3, and 5 for generating parity - c hec k matrices of LDPC co des. These degree distributions are f r o m the edge p ersp ectiv e, and they are obtained from the densit y ev olution optimization [16]. ————— ————— — ——————————————————————————— Co de 1: 1 − R 2 = 0 . 876 λ ( x ) = 0 . 3424 x + 0 . 165 x 2 + 0 . 1203 x 4 + 0 . 0191 x 5 + 0 . 012 x 6 + 0 . 1416 x 10 + 0 . 0211 x 25 + 0 . 0202 x 26 + 0 . 0185 x 34 + 0 . 0428 x 36 + 0 . 0133 x 38 + 0 . 0021 x 39 + 0 . 0104 x 40 + 0 . 0704 x 99 ρ ( x ) = 0 . 8 x 3 + 0 . 2 x 4 ————— ————— — ——————————————————————————— Co de 2: 1 − R 2 = 0 . 863 λ ( x ) = 0 . 3424 x + 0 . 165 x 2 + 0 . 1203 x 4 + 0 . 0191 x 5 + 0 . 012 x 6 + 0 . 1416 x 10 + 0 . 0211 x 25 + 0 . 0202 x 26 + 0 . 0185 x 34 + 0 . 0428 x 36 + 0 . 0133 x 38 + 0 . 0021 x 39 + 0 . 0104 x 40 + 0 . 0704 x 99 ρ ( x ) = 0 . 8 x 3 + 0 . 2 x 4 ————— ————— — ——————————————————————————— Co de 3: 1 − R 2 = 0 . 843 λ ( x ) = 0 . 3151 x + 0 . 1902 x 2 + 0 . 045 x 4 + 0 . 1705 x 6 + 0 . 1405 x 17 + 0 . 0081 x 37 + 0 . 044 x 41 + 0 . 0863 x 66 ρ ( x ) = 0 . 5 x 3 + 0 . 5 x 4 ————— ————— — ——————————————————————————— Co de 4: 1 − R 2 = 0 . 828 λ ( x ) = 0 . 3038 x + 0 . 1731 x 2 + 0 . 0671 x 4 + 0 . 0123 x 5 + 0 . 1341 x 6 + 0 . 0314 x 12 + 0 . 0108 x 14 + 0 . 0256 x 16 + 0 . 0911 x 19 + 0 . 04 x 39 + 0 . 0117 x 51 + 0 . 0189 x 57 + 0 . 0112 x 62 + 0 . 0684 x 76 ρ ( x ) = 0 . 2 x 3 + 0 . 8 x 4 ————— ————— — ——————————————————————————— Co de 5: 1 − R 2 = 0 . 818 λ ( x ) = 0 . 3038 x + 0 . 1731 x 2 + 0 . 0671 x 4 + 0 . 0123 x 5 + 0 . 1341 x 6 + 0 . 0314 x 12 + 0 . 0108 x 14 + 0 . 0256 x 16 + 0 . 0911 x 19 + 0 . 04 x 39 + 0 . 0117 x 51 + 0 . 0189 x 57 + 0 . 0112 x 62 + 0 . 0684 x 76 ρ ( x ) = 0 . 2 x 3 + 0 . 8 x 4 ————— ————— — ——————————————————————————— Co de 11: 1 − R 2 = 0 . 294 λ ( x ) = 0 . 1392 x + 0 . 2007 x 2 + 0 . 2522 x 6 + 0 . 0134 x 11 + 0 . 171 x 17 + 0 . 0424 x 31 + 0 . 0855 x 41 + 0 . 0953 x 49 ρ ( x ) = 0 . 3 x 16 + 0 . 7 x 17 ————— ————— — ——————————————————————————— Co de 13: 1 − R 2 = 0 . 92 λ ( x ) = 0 . 4051 x + 0 . 1716 x 2 + 0 . 0995 x 4 + 0 . 0447 x 5 + 0 . 0379 x 6 + 0 . 0612 x 10 + 0 . 0189 x 14 + 0 . 0333 x 16 + 0 . 0026 x 17 + 0 . 0128 x 20 + 0 . 0435 x 28 + 0 . 0075 x 50 + 0 . 0123 x 52 + 0 . 0258 x 62 + 0 . 0065 x 63 + 0 . 0166 x 71 ρ ( x ) = 0 . 4 x 2 + 0 . 6 x 3 18 ————— ————— — ——————————————————————————— Co de 14: 1 − R 2 = 0 . 605 λ ( x ) = 0 . 2366 x + 0 . 31 38 x 3 + 0 . 07 1 5 x 4 + 0 . 1707 x 10 + 0 . 0005 x 11 + 0 . 00 0 2 x 18 + 0 . 0002 x 19 + 0 . 0002 x 20 + 0 . 0003 x 21 + 0 . 0007 x 22 + 0 . 0183 x 23 + 0 . 1854 x 24 + 0 . 0016 x 25 ρ ( x ) = 0 . 9 x 6 + 0 . 1 x 7 ————— ————— — ——————————————————————————— F or the parit y-c hec k matrices of the LDPC co des in T ables 2 and 4 w e hav e used (9 , 10), (7 , 8), and (3 , 10)-regular co des. No t e that, in order to get the exact rate of 1 − R 2 , some ro ws of the parit y-c hec k matrices are randomly remo ved. 19