Asymptotic Concentration Behaviors of Linear Combinations of Weight Distributions on Random Linear Code Ensemble

1 Asymptotic Concentration Beha viors of Linear Combinations of W eight Distrib utions on Random Linear Code Ensemble T adashi W adayama Abstract — Asymptotic co ncentration beha viors of li near com- binations of weight distributions on the random linear code ensemble are presented. Many important properties of a bin ary linear code can be expressed as the form of a l inear combination of weight distributions su ch as number of codewords, undetected error probability and upp er b ound on the maximum l ikelihood error probability . The key i n th is analysis is the cov ariance fo rmula of weight distributions of the rand om linear code ensemble, which rev eals the second-order statistics of a lin ear function of the weight distributions. Based on the cov ariance fo rmula, sev eral expressions of the asymptotic concentration rate , which ind icate the speed of conv er gence to t he av erage, are derive d. I . I N T RO D U C T I O N For a b inary random c ode ensemble or a b inary rando m linear co de ensemble, th e asympto tic behaviors of the first moment (expectation) of some properties of interest ha ve been studied extensively . For exam ple, the erro r expon ent derived by Gallager [1] is a celebrated consequence of such a first-moment analysis. R ecent advances in second-mo ment analysis on low-density parity check m atrix ensemb les [5], [6] h av e encou raged studies on the secon d-ord er behaviors (fluctuation fr om the av erage) of the m acroscopic pro perties of an ensemble, wh ich ha d p reviously attracted little attention . In this paper, asym ptotic co ncentration behaviors of linear combinatio ns of weight distributions on the random linear code ensemble are presented. Many important pro perties of a binary linear cod e can be expressed as the f orm o f a linear c ombinatio n of weight d istributions such as n umber o f codewords, undetected error probab ility and upper boun d on the maximum likelihood (ML) error probab ility . The key in this analysis is the covariance fo rmula o f weight distributions of the r andom linear code ensemb le, which reveals the second- order statistics of a linear fun ction o f the weig ht distributions. Based on the cov ariance form ula, se veral expressions of the asymptotic concen tration rate , which indicate the speed of conv ergence to the average, are deri ved. I I . P R E L I M I N A R I E S A. Ensemble, expectation, an d covariance Let G be a set of bina ry m × n m atrices where m a nd n are positive integers. Sup pose th at proba bility P ( H ) is assigned for each matrix H in G , where P ( H ) is a p robab ility ma ss T . W aday ama is with Department of Computer Science , Nagoya Institute of T ech nology , Nagoya, 466-8555, Japan (e-mail: wadayama@n itech.a c.jp) function defined on G such that P H ∈G P ( H ) = 1 , and ∀ H ∈ G , P ( H ) > 0 . T he pair {G , P ( H ) } ca n b e consider ed as an ensemble of matrices . Althou gh it is an abuse of notation, for simplicity , we will not d istinguish {G , P ( H ) } from G . Let f ( · ) be a re al-valued f unction d efined on G , which can b e consid ered as a random variab le . The expec tation of f ( · ) with respect to the ensemble G is defined by E G [ f ] △ = P H ∈G P ( H ) f ( H ) . The v ariance of f ( · ) is given by V AR G [ f ] △ = E G [ f ( H ) 2 ] − E G [ f ( H )] 2 . In a similar way , the cov ariance between two rea l-valued functions f ( · ) , g ( · ) defined on G is given b y COV G [ f , g ] △ = E G [ f g ] − E G [ f ]E G [ g ] . (1) Let { g 1 ( · ) , g 2 ( · ) , . . . , g n ( · ) } be a set of real-valued functions defined o n G , and let f ( · ) be a linear combin ation of g i ( · ) : f ( H ) △ = P n i =1 φ i g i ( H ) for H ∈ G , where φ i ( i ∈ [1 , n ]) are real v alues. The notatio n [ a, b ] denotes the set of consecutive integers f rom a to b . It is easy to show that th e variance o f f ( · ) is given b y V AR G [ f ] = n X i =1 n X j =1 φ i φ j COV G [ g i , g j ] , (2) e.g., see [7] for details. B. W eigh t distribution The we ight d istributions { A 1 ( · ) , . . . , A n ( · ) } , which can be considered a s a set o f r eal-valued function s defined on G , is defined by A w ( H ) △ = X x ∈ Z ( n,w ) I [ H x = 0 m ] , w ∈ [0 , n ] , (3) for any H ∈ G , where Z ( n,w ) denotes the set o f all bin ary n - tuples with weight w . The function I [ · ] is the indicator f unction such th at I [ condition ] = 1 if condition is true; otherwise, it giv es 0 . In the p resent paper, sym bol shown in bold , such as x , de note c olumn vector s. Let C ( H ) be the binary linear code defin ed based o n H , namely , C ( H ) △ = { x ∈ F n 2 : H x = 0 m } , where F 2 denotes the binary Galo is field. Many prop erties of C ( H ) of inter est can b e rep resented by a linear combina tion of the weight distributions { A w ( · ) } n w =1 . Let F ( · ) be such a p roperty of C ( H ) , which is expressed as F ( H ) △ = P n w =1 Φ w A w ( H ) for any H ∈ G , where Φ w ( w ∈ [0 , n ]) are re al values. 2 For example, the un detected error p robability of C ( H ) can be expressed as a linear com bination of the weight distributions of C ( H ) when it is u sed as an error detection code for a bin ary sym metric channel (BSC). The expression is g i ven by F ( H ) = P n w =1 A w ( H ) ǫ w (1 − ǫ ) n − w , wh ere ǫ is the crossover prob ability of the BSC. In th is setting, th e prope rty F ( · ) ca n be regarded as a random variable tha t takes a real value. I t is natura l to stud y its statistics su ch as expe ctation, variance for a given ensemble of b inary matr ices. C. Rando m linea r co de en semble In the present pa per, we deal with an en semble of binary matrices, wh ich is called the random line ar code ensemble . Definition 1 : The rand om line ar code ensem ble R n,m con- tains all b inary m × n matr ices. Equ al prob ability P ( H ) = 1 / 2 nm is assigne d fo r each matrices in R n,m . Note that although the random linear code ensemble is actually an e nsemble of matrices, it is regarded her ein as an en semble of b inary linea r cod es. The expectation of weigh t distributions of rando m ensemble is known [2] to be E R n,m [ A w ] =  n w  2 − m for n ≥ 1 . The next theorem provid es a clo sed form ula of th e covariance of weigh t distributions over the r andom lin ear co de en semble. Theor em 1: Assume a r andom ensem ble R n,m . Th e cov ari- ance o f A w 1 ( · ) and A w 2 ( · ) is given b y COV R n,m [ A w 1 , A w 2 ] =  0 , 0 < w 1 , w 2 ≤ n, w 1 6 = w 2 (1 − 2 − m )2 − m  n w  , 0 < w 1 = w 2 ≤ n. (4) (Proof) The pro of is gi ven in A ppendix . The variance of the weight distrib utions of the random lin- ear cod e ensemble has alrea dy been shown in [4]. T hus, the new con tribution of this theore m is th e case in which COV R n,m ( A w 1 , A w 2 ) = 0 when w 1 6 = w 2 . This theorem implies th at the p air o f r andom variables A w 1 and A w 2 ( w 1 6 = w 2 ) is pairwise independ ent 1 . I I I . F O R M U L A S O N A S Y M P T OT I C C O N C E N T R A T I O N R AT E A. Asymptotic b ehaviors of e xpectation Definition 2 : Let G n be an ensemble of bina ry (1 − R ) n × n matrices. Th e parameter R , called the design rate, is a real value in the r ange of 0 < R < 1 . Suppose that f ( · ) is a real-valued f unction defined on G . The asymptotic exponent of E G [ f ] is given by ξ △ = lim n →∞ 1 n log E G n [ f ] (5) if the limit exists. Namely , asymptotically , E G [ f ] behav es like E G [ f ( H )] . = 2 ξn where the notation a n . = b n means th at lim n →∞ (1 /n ) lo g a n = lim n →∞ (1 /n ) lo g b n . In the pr esent paper, a logarithm of base 2 is denoted by log . 1 Note that the set of random v ariabl es { A 1 , . . . A n } are not m utuall y indepen dent because P n w =1 A w ( H ) ≥ 2 n − m − 1 holds for any instanc e H in R n,m . In the case of th e rando m linear co de ensemble, it has b een reported [2 ] that lim n →∞ 1 n log E R n, (1 − R ) n [ A θ n ] = H ( θ ) − (1 − R ) , (6) holds for 0 < θ ≤ 1 , where H ( · ) is the binar y entropy function defined by H ( x ) △ = − x log x − (1 − x ) log (1 − x ) . The p arameter θ is called the normalized weigh t . B. Asymptotic co ncentration rate As th e size of th e matrix goes to infin ity , the value of f ( · ) is often sharply con centrated around its expectation . The asymptotic co ncentration rate is defined as follows. Definition 3 : Let G n be an ensemb le of bin ary (1 − R ) n × n matrices, wh ere R is a r eal v alue in the r ange of 0 < R < 1 . For a real-valued f unction f ( · ) defined on G n , the asy mptotic concentr ation rate ( abbreviated as A CR) of f ( · ) is d efined by η △ = lim n →∞ 1 n log  V AR G n [ f ] E G n [ f ] 2  . (7) if the limit exists. The following lemma e xplains the importance of the asymp - totic con centration rate. Lemma 1: Let η be the asymptotic concentr ation rate of f ( · ) . For any positive real number α , lim n →∞ 1 n log P r  f ( H ) E G n [ f ] / ∈ (1 − α, 1 + α )  ≤ η (8) holds if E G n [ f ] > 0 for any sufficiently large n . (Proof) Based on the Cheby shev ineq uality , th e ineq uality P r h | f ( H ) − E G n [ f ] | > c p V AR G n [ f ] i ≤ 1 c 2 (9) holds for any real numbe r c > 0 . Suppose that c is g iv en by c = α E G n [ f ] p V AR G n [ f ] . (10) where α is a po siti ve real nu mber . Fr om the assumption E G n [ f ] > 0 , it is easy to verify that c becom es positiv e. Substituting (1 0) in to (9), we h av e P r [ | f ( H ) − E G n [ f ] | > α E G n [ f ]] ≤ V AR G n [ f ] α 2 E G n [ f ] 2 . (11) Due to th e assump tion E G [ f ( H )] > 0 , the above in equality can b e rewritten in the f ollowing form: P r  f ( H ) E G n [ f ] / ∈ (1 − α, 1 + α )  ≤ V AR G n [ f ] α 2 E G [ f ] 2 . (12) Considering the asymp totic expone nt o f th e above equa tion, we obtain the claim of the lem ma. From the asymp totic co ncentration rate , we c an clarif y the probab ilistic conv ergence behavior of f ( · ) . If η < 0 ho lds, f ( H ) /E G n [ f ] c onv erges to 1 in pro bability as n g oes to infinity . This mean s that η < 0 is a sufficient condition of the conv ergence in p robability . The asymp totic concentr ation rate ind icates the speed of this con vergence 3 Example 1: The variance o f th e weight distributions of the random linear code ensemble is given by V AR R n, (1 − R ) n [ A θ n ] = (1 − 2 − (1 − R ) n )2 − (1 − R ) n  n θn  . (13) Therefo re, the asymptotic exponen t of the v ariance beco mes lim n →∞ 1 n log V AR R n, (1 − R ) n [ A θ n ] = H ( θ ) − (1 − R ) . (14) From th is expon ent, we imm ediately have the asymptotic concentr ation rate of the weight distribution: η = lim n →∞ 1 n log V AR R n, (1 − R ) n [ A θ n ] E R n, (1 − R ) n [ A θ n ] 2 = H ( θ ) − (1 − R ) − 2 ( H ( θ ) − (1 − R )) = 1 − R − H ( θ ) . (15) Let the min imum roo t of equation 1 − R − H ( θ ) = 0 be θ GV , which is called the r elative Gilbert-V ar shamov ( GV) distance . Since η < 0 ho lds in th e range θ GV < θ < 1 − θ GV , A θ n ( H ) /E R n, (1 − R ) n [ A θ n ] co n verges to 1 in pro bability as n goes to infinity [3]. C. AC R of a lin ear combination of weight distributions The goal of th e present paper is to observe the asymptotic behavior of the variance of the linear combina tion defined in (16) o f the weight distrib utions: F ( H ) = n X w =1 Φ w A w ( H ) . (16) The next th eorem gives the asymp totic con centration rate of F ( H ) . Theor em 2: Let G n be an ensemble of binary (1 − R ) n × n matrices, wh ich have th e following asymptotic first- and second-o rder b ehaviors: E G n [ A θ n ] . = 2 n ( H ( θ )+ q ( θ )) , (17) COV G n [ A θ 1 n , A θ 2 n ] . = 2 nγ ( θ 1 ,θ 2 ) . (18) The asymptotic con centration rate of F ( · ) define d in (1 6) is giv en by η = sup 0 <θ 1 ≤ 1 sup 0 <θ 2 ≤ 1 [ φ ( θ 1 ) + φ ( θ 2 ) + γ ( θ 1 , θ 2 )] − 2 sup 0 <θ ≤ 1 [ φ ( θ ) + H ( θ ) + q ( θ )] , (19) where φ ( θ ) is defined by φ ( θ ) △ = lim n →∞ 1 n log Φ θ n . (20) (Proof) It is easy to verify that lim n →∞ 1 n log E G n [ F ( H )] = sup 0 <θ ≤ 1 [ φ ( θ ) + H ( θ ) + q ( θ )] (21) holds. Using Eq.(2), we h av e lim n →∞ 1 n log V AR G n [ F ] = sup 0 <θ 1 ≤ 1 sup 0 <θ 2 ≤ 1 [ φ ( θ 1 ) + φ ( θ 2 ) + γ ( θ 1 , θ 2 )] . (22) Substituting (2 1) and (22) into the definition of the AC R, the theorem is proven. The next corollar y is a special case of th e above theor em for the rando m linear code ensem ble. Cor olla ry 1 : The AC R of F ( · ) de fined in ( 16) over the random linear code ensemble R n, (1 − R ) n is g i ven by η = sup 0 <θ ≤ 1 [2 φ ( θ ) + H ( θ )] − sup 0 <θ ≤ 1 [2 φ ( θ ) + 2 H ( θ )] + 1 − R, (23) where φ is giv en in ( 20). (Proof) In the case of the random ensem ble, q ( θ ) is given by q ( θ ) = − (1 − R ) for 0 < θ ≤ 1 . From Theo rem 1, we c an derive the exponent of the covariance γ ( θ 1 , θ 2 ) , which is g iv en by γ ( θ 1 , θ 2 ) =  −∞ , θ 1 6 = θ 2 H ( θ ) − (1 − R ) , θ 1 = θ 2 , (24) where 0 < θ 1 , θ 2 ≤ 1 . Pluggin g these fun ctions into the formu la in Theore m 2, we o btain the claim of the c orollary . Example 2: In this example, we will discuss th e num - ber o f co dew ords in C ( H ) . L et u s define M ( H ) △ = 1 + P n w =1 A w ( H ) , which is th e number of co dewords of C ( H ) . In this case, we can see that Φ w = 1 holds for 1 ≤ w ≤ n . The asy mptotic expon ent of M ( H ) is g iv en by lim n →∞ 1 n log E R n,m [ M ] = sup 0 <θ ≤ 1 [ H ( θ )] − (1 − R ) = R. (25) From th e d efinition of M ( H ) , we immediately h av e φ ( θ ) = 0 , 0 < θ ≤ 1 . Using Co rollary 1, we obtain η = sup 0 <θ ≤ 1 [ H ( θ )] − sup 0 <θ ≤ 1 [2 H ( θ )] + 1 − R = − R. (26) Since R is a positive real number, M ( H ) / E R n,m [ M ] con- verges to 1 in pro bability for any R > 0 . In some cases, the asymp totic con centration r ate ca n b e written in a closed from w ithout an optim ization p rocess required in Corollary 1. Theor em 3: Assume the rand om lin ear code ensemble with design r ate R . Let K 1 , K 2 be real positive constants that d o not depen d on n . If Φ w is expr essed as Φ w = K w 1 K n − w 2 , then the ACR of F ( H ) = P n w =1 Φ w A w ( H ) is given by η = log K 2 1 + K 2 2 ( K 1 + K 2 ) 2 + 1 − R. (27) (Proof) Using T heorem 1 an d the b inomial theorem , we hav e V AR R n, (1 − R ) n [ F ] = n X w 1 =1 n X w 2 =1 ( K w 1 + w 2 1 K 2 n − w 1 − w 2 2 )COV R n, (1 − R ) n [ A w 1 , A w 2 ] = n X w =1 ( K 2 w 1 K 2 n − 2 w 2 )(1 − 2 − m )2 − m  n w  = (1 − 2 − m )2 − m n X w =0  n w  ( K 2 1 ) w ( K 2 2 ) n − w ! 4 − (1 − 2 − m )2 − m K 2 n 2 = (1 − 2 − m )2 − m  K 2 1 + K 2 2  n − (1 − 2 − m )2 − m K 2 n 2 . (2 8) Thus, th e asympto tic exponen t of V AR R n, (1 − R ) n [ F ] is given by lim n →∞ 1 n log V AR R n, (1 − R ) n [ F ] = log  K 2 1 + K 2 2  − (1 − R ) . (29) In a similar way , E R n, (1 − R ) n [ F ] can be r ewritten as follows: E R n, (1 − R ) n [ F ] = n X w =1 ( K w 1 K n − w 2 )E R n, (1 − R ) n [ A w ] = 2 − m n X w =0 ( K w 1 K n − w 2 )  n w  ! − 2 − m K n 2 = 2 − m ( K 1 + K 2 ) n − 2 − m K n 2 . (30) This leads to the expone nt of the expectation: lim n →∞ 1 n log E R n, (1 − R ) n [ F ] = log ( K 1 + K 2 ) − (1 − R ) . (31 ) Substituting the ab ove two equa tions into the definition of the A CR, we hav e the claim o f the theorem . Example 3: Assume the binary symmetric channel with crossover pro bability ǫ . The undetected error probability of C ( H ) is given b y P U ( H ) = P n w =1 A w ( H ) ǫ w ǫ n − w . In this case, the error exponent becom es lim n →∞ − 1 n E R n, (1 − R ) n [ P U ] = 1 − R. (32) Since Φ w = ǫ w ǫ n − w has the form stated in Theor em 3 (i.e., K 1 = ǫ, K 2 = 1 − ǫ ), we can apply T heorem 3 an d o btain η = log( ǫ 2 + (1 − ǫ ) 2 ) + 1 − R . This results suggests the existence of the con vergence threshold ǫ ∗ for gi ven R such that ǫ ∗ separates the co ncentratio n regime a nd the n on-co ncentration regime o f ǫ . T he r oot of lo g ( ǫ 2 + (1 − ǫ ) 2 ) + 1 − R = 0 becom es an uppe r bound of ǫ ∗ . Let ǫ ′ be the roo t of th e e quation lo g( ǫ 2 + (1 − ǫ ) 2 ) + 1 − R = 0 . T able I p resents some values of ǫ ′ for 0 . 1 ≤ R ≤ 0 . 9 . When ǫ > ǫ ′ , we ha ve log( ǫ ′ 2 + (1 − ǫ ′ ) 2 )+ 1 − R < 0 . In such a r egion, P U ( · ) co ncentrates ar ound its average value in the limit as n tends to infinity . T ABLE I R O O T S O F log( ǫ 2 + (1 − ǫ ) 2 ) + 1 − R = 0 R ǫ ′ 0.1 0.366047 0.2 0.307193 0.3 0.259613 0.4 0.217375 0.5 0.178203 0.6 0.140933 0.7 0.104872 0.8 0.069564 0.9 0.034687 I V . AC R O F T H E U P P E R B O U N D O F M L E R RO R P R O BA B I L I T Y A. Bhattacharya boun d In the following discussion, the binary symmetric channel with cr ossover probability ǫ is assumed for simplicity . Assume that M L d ecoding is u sed in a deco der . For a binar y m × n parity ch eck matrix H , the block er ror pr obability P e ( H ) can be u pper bo unded as follows: P e ( H ) ≤ n X w =1 A w ( H ) D w , where D is c alled the Bhattachary a para meter an d is defined as D △ = 2 p ǫ (1 − ǫ ) . The up per bo und is called the Bh attacharya bound [1 ] an d has the form of a linear comb ination of weight distributions. Let us define B ( H ) △ = P n w =1 A w ( H ) D w . It is expected that the statistics of B ( H ) reflects the asymp totic b ehavior of ac tual ML pr obability o f an ensemble. W e fir st derive th e asym ptotic expression of the er ror exp o- nent of the Bhattacharya bound in the case o f the random linear code ensemble. Th e expectatio n of B ( H ) has the following closed f orm expression : E R n, (1 − R ) n [ B ] = n X w =1 E R n, (1 − R ) n [ A w ( H )] D w = n X w =1  n w  2 − (1 − R ) n  2 p ǫ (1 − ǫ )  w = 2 − (1 − R ) n (2 p ǫ (1 − ǫ ) + 1 ) n − 2 − (1 − R ) n . Thus, the error exponent of E R n, (1 − R ) n [ B ] is giv en by lim n →∞ − 1 n log E R n, (1 − R ) n [ B ] = 1 − R − log  2 p ǫ (1 − ǫ ) + 1  . (33) This is a part of the error exponen t fun ction derived b y Gallager [1] ( see also [3]) in the low-rate regime 2 . Namely , the Bhattacharya bound correspon ds to th e u pper bound d ue to Gallage r with the parameter ρ = 1 [1]. In the following, we will examine th e asym ptotic co ncen- tration rate of the Bhattach arya bo und. Cor olla ry 2 : The A CR of B ( H ) is gi ven by η = log 4 ǫ ( ǫ − 1) + 1 (2 p ǫ (1 − ǫ ) + 1) 2 ! + 1 − R. (34) (Proof) By letting K 1 = D an d K 2 = 1 and using Th eorem 3, we obtain η = lo g  ( D 2 + 1) / ( D + 1) 2  + 1 − R. Substituting D = 2 p ǫ (1 − ǫ ) into this eq uation, the coro llary is proven. B. Expur gated boun d W e here con sider the expu rgated ensemble R ∗ n, (1 − R ) n , which can be o btained f rom R n, (1 − R ) n by expurgating parity check matrices with A θ n ( H ) 6 = 0 for 0 < θ < θ GV , 1 − θ GV < θ ≤ 1 . The asymp totic g rowth rate of the weight distributions is the same for the origin al an d expurgated e nsembles when θ GV ≤ θ ≤ 1 − θ GV . Howe ver , q ( θ ) becomes −∞ when 2 It has been reported that thi s exponent is asymptoti cally tight if R x ≤ R ≤ R cr it [3]. 5 0 < θ < θ GV , 1 − θ GV < θ ≤ 1 in the case o f the expurgated ensemble. The er ror exponent o f E R ∗ n, (1 − R ) n [ B ] is gi ven by lim n →∞ − 1 n log E R ∗ n, (1 − R ) n [ B ] = min θ GV ≤ θ ≤ 1 − θ GV { 1 − R − H ( θ ) − θ log(2 p ǫ (1 − ǫ )) } . If θ cr it ≥ θ GV , the minimum in the ab ove equation is attained at θ = θ cr it , where θ cr it △ = 2 p ǫ (1 − ǫ ) 1 + 2 p ǫ (1 − ǫ ) . (35) In this case, the expo nent coincides with the exponent given in Eq.(3 3). Othe rwise, ( θ cr it < θ GV ) , the min imum occurs at θ = θ GV . Therefore, we hav e lim n →∞ − 1 n log E R ∗ n, (1 − R ) n [ B ] = − θ GV log(2 p ǫ (1 − ǫ )) . (36) if θ cr it < θ GV . T his expo nent corr esponds to the usua l expur ga ted exponent for th e BSC case (see also th e discussion in [ 3]). Th e next corollary states the A CR of the u pper bound of ML erro r prob ability in the case of θ cr it < θ GV : Cor olla ry 3 : If θ cr it < θ GV , the ACR is given b y η = 0 . (Proof) Since the expurgated ensemble can be obtained from the or iginal ensemb le by rem oving a sub-exp onential n umber of matrice s, the expon ent of the variance, i.e ., γ ( θ 1 , θ 2 ) , takes the same v alues for th e original and expurgated en sembles if θ GV ≤ θ 1 , θ 2 ≤ 1 − θ GV . From Theo rem 2, we have η = max θ GV ≤ θ ≤ 1 − θ GV h H ( θ ) + 2 θ log(2 p ǫ (1 − ǫ )) i − max θ GV ≤ θ ≤ 1 − θ GV h 2 H ( θ ) + 2 θ log(2 p ǫ (1 − ǫ )) i + 1 − R because q ( θ ) = −∞ for θ < θ GV in the case of the expurgated ensemb le. Fro m the assumption θ cr it < θ GV , 2 H ( θ ) + 2 θ log (2 p ǫ (1 − ǫ )) is maximized a t θ = θ GV . Note that − H ( θ GV ) + 1 − R = 0 ho lds. Mor eover , H ( θ ) + 2 θ log(2 p ǫ (1 − ǫ )) is also maximized at θ = θ GV . A P P E N D I X 1) Pr eparation of the pr oof of Theo r em 1: The second moment of the weight distribution for a giv en ensemble G is g i ven by E G [ A w 1 A w 2 ] = E G   X x ∈ Z ( n,w 1 ) X y ∈ Z ( n,w 2 ) I [ H x = 0 m , H y = 0 m ]   = X x ∈ Z ( n,w 1 ) X y ∈ Z ( n,w 2 ) E G [ I [ H x = 0 m , H y = 0 m ]] . (37) For the case in which G = R n,m , we obta in E R n,m [ A w 1 A w 2 ] = X x ∈ Z ( n,w 1 ) X y ∈ Z ( n,w 2 ) # { H : H x = 0 m , H y = 0 m } 2 mn . (38) Here, we encounte r a p roblem o f c ounting th e m atrices that satisfy both H x = 0 m and H y = 0 m . Before solving this counting pr oblem, we first introduce some notation. Suppose that w 1 > 0 an d w 2 > 0 . For a given p air ( x , y ) ∈ Z ( n,w 1 ) × Z ( n,w 2 ) , the index sets I 1 , I 2 , I 3 , and I 4 are defined as follows: I 1 △ = { k ∈ [1 , n ] : x k = 1 , y k = 0 } , I 2 △ = { k ∈ [1 , n ] : x k = 1 , y k = 1 } , I 3 △ = { k ∈ [1 , n ] : x k = 0 , y k = 1 } , I 4 △ = { k ∈ [1 , n ] : x k = 0 , y k = 0 } , where x = ( x 1 , x 2 , . . . , x n ) a nd y = ( y 1 , y 2 , . . . , y n ) . The size o f each index set is den oted by i k = # I k ( k = 1 , 2 , 3 , 4) . Let h = ( h 1 , h 2 , . . . , h n ) t be a binary n -tuple (a row vector). The par tial weigh t of h cor respond ing to an index set I k ( k = 1 , 2 , 3 , 4 ) is de noted by w k ( h ) , namely , w k ( h ) △ = # { j ∈ I k : h j = 1 } . Since th e ind ex sets are mutually exclusi ve, th e eq uation i 1 + i 2 + i 3 + i 4 = n holds and i 2 can take the integer values in the following range: max { w 1 + w 2 − n, 0 } ≤ i 2 ≤ min { w 1 , w 2 } . The size of each in dex set can be expressed as i 1 = w 1 − i 2 , i 3 = w 2 − i 2 , i 4 = n − ( w 1 + w 2 − i 2 ) . The next lemm a forms the basis for the proof of Theorem 1. Lemma 2: For any x ∈ Z ( n,w 1 ) and y ∈ Z ( n,w 2 ) (0 < w 1 , w 2 ≤ n ) , the fo llowing eq ualities ho ld: # { h ∈ F n 2 : hx = 0 , hy = 0 } =  2 n − 2 x 6 = y , 2 n − 1 x = y . (39) (Proof) In the following, we prove the lemm a for the con di- tions 0 < w 1 ≤ w 2 ≤ n . The proo f for the o pposite case 0 < w 2 ≤ w 1 ≤ n then follows immed iately up on exchangin g the variables w 2 and w 1 in the pr oof. First, we will sho w that # { h ∈ F n 2 : hx = 0 , hy = 0 } = 2 n − 2 (40) if 0 < w 1 ≤ w 2 ≤ n and x 6 = y . L et the support sets o f x an d y b e S ( x ) △ = { i ∈ [1 , n ] : x i = 1 } an d S ( y ) △ = { i ∈ [1 , n ] : y i = 1 } , respectively . The f ollowing three cases should be trea ted separ ately: • Case (i) : 0 < i 2 < w 1 (i.e., S ( x ) a nd S ( y ) overlap but S ( y ) does no t inclu de S ( x ) .) • Case (ii): i 2 = 0 (i.e., S ( x ) a nd S ( y ) do no t overlap.) • Case (iii): i 2 = w 1 (i.e., S ( y ) in cludes S ( x ) .) First, we con sider Case (i). From th e assumptio n that 0 < i 2 < w 1 , it is clear tha t I 1 6 = ∅ (b ecause i 2 < w 1 ), I 2 6 = ∅ (because i 2 > 0 ), I 3 6 = ∅ (because w 2 ≥ w 1 > i 2 ). For any h ∈ F n 2 , th e e quations hx t = 0 and hy t = 0 hold if an d on ly if w i ( h ) is ev en fo r i = 1 , 2 , 3 or w i ( h ) is odd fo r i = 1 , 2 , 3 . Thus, the n umber of vector s satisfying the above condition is giv en by N h = 2 × 2 i 1 − 1 × 2 i 2 − 1 × 2 i 3 − 1 × 2 i 4 = 2 n − 2 , (41) where N h is d efined by N h △ = # { h ∈ F n 2 : hx t = 0 , hy t = 0 } . In the ab ove deriv a tion, we used the eq ualities: w 1 = i 1 + i 2 , w 2 = i 2 + i 3 , i 4 = n − ( w 1 + w 2 − i 2 ) . Note that Eq. (41) (an d Eqs. (42, )(43), and ( 44) to be presen ted below) holds regardless of the size of I 4 ( i 4 = 0 or i 4 > 0 ). 6 W e now con sider Case (ii). F or this case, I 1 6 = ∅ ( since w 1 > 0 ), I 2 = ∅ (since i 2 = 0 ) an d I 3 6 = ∅ (since w 2 > 0 ). The equalities hx = 0 an d hy = 0 h old if an d on ly if w i ( h ) is even fo r i = 1 , 3 holds. The nu mber of vectors satisfying the conditio n is giv en by N h = 2 i 1 − 1 × 2 i 3 − 1 × 2 i 4 = 2 n − 2 . (42) The final case is Case (iii). For this c ase, I 1 = ∅ (since i 2 = w 1 ), I 2 6 = ∅ ( since i 2 = w 1 > 0 ) and I 3 6 = ∅ ( since x 6 = y an d w 1 ≤ w 2 ). T hese cond itions lead to the conditio n: w i ( h ) is e ven f or i = 2 , 3 for hx = 0 , hy = 0 . Again, 2 n − 2 n -tuples satisfy the above con dition, namely , N h = 2 i 2 − 1 × 2 i 3 − 1 × 2 i 4 = 2 n − 2 . (43) Combining the above results fo r Cases (i), ( ii), and (iii), we obtain Eq. (40). W e then show th at N h = 2 n − 1 holds if 0 < w 1 = w 2 ≤ n and x = y . For th is case, we h ave I 1 = ∅ , I 2 6 = ∅ , I 3 = ∅ (since x = y ) . T hus, th e eq uations hx = 0 , hy = 0 ho ld if and on ly if w 2 ( h ) is even . The numb er o f n -tu ples satisfying the ab ove condition is given by N h = 2 i 2 − 1 × 2 i 4 = 2 n − 1 . (44) The p roof o f this lemma is com pleted. 2) Pr oo f of Theorem 1: Th e p roof of Theor em 1 consists of two parts. Th e first part co rrespond s to the case in which the covariance bec omes zero. T he second part c orrespon ds to the case in which the covariance bec omes non -zero. W e co mmence with the first p art o f th e pr oof. Assume that 0 < w 1 , w 2 ≤ n, x 6 = y . From Lemma 2, we o btain # { H : H x = 0 m , H y = 0 m } = m Y k =1 # { h ∈ F n 2 : hx = 0 , hy = 0 } = 2 m ( n − 2) . (45) Substituting into (38), we o btain E R n,m [ A w 1 A w 2 ] = X x ∈ Z ( n,w 1 ) X y ∈ Z ( n,w 2 ) 2 m ( n − 2) 2 mn = 2 − 2 m  n w 1  n w 2  = E R n,m [ A w 1 ]E R n,m [ A w 2 ] . (46) The last equ ality is equiv alent to COV R n,m [ A w 1 , A w 2 ] = 0 . W e n ow c onsider the second p art o f the pro of: Assume that x = y . From Lemma 2 , we have # { H : H x t = 0 , H y t = 0 } = 2 m ( n − 1) , and E R n,m [ A 2 w ] = X x ∈ Z ( n,w ) X y ∈ Z ( n,w ) I [ x = y ]2 m ( n − 1) 2 mn + X x ∈ Z ( n,w ) X y ∈ Z ( n,w ) I [ x 6 = y ]2 m ( n − 2) 2 mn = 2 − m  n w  + 2 − 2 m  n w  n w  −  n w  = E R n,m [ A w ] 2 + 2 − m  n w  − 2 − 2 m  n w  . (47) The last equality is equivalent to COV R n,m ( A w , A w ) = (1 − 2 − m )2 − m  n w  . A C K N O W L E D G M E N T The present study was suppor ted in part by the M inistry of Education , Science, Sports, and Culture of Japan thro ugh a Gran t-in-Aid f or Scientific Research on Prior ity Areas (Deepenin g and Expansion of Statistical Informatics) No . 18079 0091 . R E F E R E N C E S [1] R.G.Gallage r , ”Informati on Theory and Reliable Communic ation” . J ohn W ile y & Sons, 1968. [2] R.G.Gallage r , ”Low Density P arity Chec k Codes” . Cambridge, MA:MIT Press 1963. [3] A.Barg, G .D.Forne y ,Jr . ”Random codes: minimum distances and err or exp onents, ” IEEE T rans. Inform. Theory , V ol.48, pp.2568–2573, No. 9, 2002. [4] T . Richard son, R. Urbanke , “Mode rn Coding Theory , ” online: http:/ /lthc www .epfl.ch/ [5] O. Barak, D. Burshtein, “Lo wer bounds on the spectrum and error rate of LDPC code ensembles, ” in Proceedings of Interna tional Symposium on Information T heory , 2005. [6] V . Rathi, “On the Asymptotic W eight Distribut ion of Regula r LDPC Ensembles, ” in Proceedings of Inter nationa l Symposium on Informat ion Theory , 2005. [7] M.Mitzen macher and Eli Upfal , ”Probability and Computing, ” Cam- bridge, 2005.