Coding Theorems for Repeat Multiple Accumulate Codes

1 Coding Theorems for Repeat Mu ltiple Accumulate Codes J ¨ org Kliewer ∗ , Kam il S. Zigangirov ‡ , Christ ian Koller ‡ , Dani el J. Costello, J r . ‡ ∗ Klipsch School of El ectrical and Computer En gineering, Ne w Mexico State Univ ersity , Las Cruces, NM 88003, USA Email: jkli e wer@nmsu.edu ‡ Department of Electrical Engineering, Univ ersity of Notre Dame, Notre Dame, IN 46556, USA Email: { kzigangi , ckoller , costello.2 } @nd.edu Abstract In this p aper the ensemble of codes formed by a serial concatena tion of a repetitio n cod e with multiple accumu lators connected through random interleavers is considered. Based on finite length weig ht enumera tors for these codes, asymptotic expressions for the minim um distan ce an d an arbitrary number of accumu lators larger than on e are derived using th e unifor m interleaver ap proach . In acco rdance with earlier results in the literature, it is first shown th at the m inimum distance of repeat- accumulate cod es can grow , at best, sublinear ly w ith block length. Then, for repeat-ac cumulate-ac cumulate co des and rates of 1 / 3 or less, it is proved that these codes exhibit asymp totically linear distance growth with block length, where the gap to th e Gilbert-V arshamov bound can be made vanishingly small by in creasing the number of accumulators b eyond two. In order to address larger rates, rando m pu ncturing of a low-rate mo ther code is intro duced. It is sh own that in th is case the resulting en semble of repea t-accumu late-accumu late codes asympto tically achiev es linear distanc e growth close to the Gilbert-V arshamov b ound. This ho lds ev en f or very high rate codes. Index T erms Multiple serial concatenatio n, repeat-accum ulate codes, unifor m interleaver , minimum distance growth rate coefficient, Gilb ert-V arshamov bou nd This work was supported in part by NSF Grants CCF-0830666 and CCF- 083065 0, NASA Grant NNX07AK53G, and the Univ ersit y of Notre Dame Faculty Research Program. Parts of the paper were presented at the 45th Annual Allerton Conference on Communication, Control, and Computing, September 2007, Monticello, IL [1]. 2 I . I N T RO D U C T I O N Since the in vention of turbo co des, s ev eral new turbo-like coding s chemes hav e be en p roposed. Among these are serially conca tenated cod es, a s imple example of which is the repe at-accumulate (RA) code [2] consisting only of a rep etition code, a n interleaver , and a n acc umulator . S uch a code has the advantage o f very low decod ing complexity compared to serially concaten ated code constructions with more c omplex constituent codes. Another benefit of RA cod es, c ompared to p owerful code constructions such as LDPC codes, is their extremely lo w encoding complexity of O (1) , wherea s LDP C cod es have an en coding complexity of O ( g ) , where g , although much smaller than the bloc k len gth [3], is greater tha n on e. This makes RA codes we ll suited in power -limi ted en vironmen ts; for example, for p hysical layer error correction in b attery powered sensor network nodes o r for sp acecraft communications . Design guidelines for achieving an interleaver gain with double serially concatenated code con structions have been given in [4] (and in [5] for more general parallel an d serially co ncatena ted code ens embles). In this pape r we add ress multiple serially co ncatena ted RA-type co des, where, in particular , we focus on the serial co ncatenation of an outer re petition c ode with two or mo re ac cumulators conne cted through rando m interleav ers. The resulting c ode ens emble is then an alyzed using the uniform interleaver ap proach [6] by av eraging over all p ossible interlea vers. Our work is mainly mo ti v ated by [7], [8], whe re a similar setup was considered . Whereas [7] co nsiders the interleaver gain of repea t multiple ac cumulate (RMA) co des (and thus address es a s pecial case of the work in [5]), in [8] the authors show that, for an asymptotically lar ge number of ac cumulators, there a re codes in the ense mble who se minimum distance grows linearly with bloc k length a nd which achieve the G ilbert-V arshamov boun d (GVB). They als o provide numerical calculations of the minimum d istance for diff erent numb ers of accumu lators and finite bloc k lengths, but they do n ot g i ve results for the asymptotic minimum distanc e growth rate co efficient in the practically more rele vant case of a finite (s mall) number o f accumulators. An extension of this work was presente d in [9], where it is s hown tha t, for a finite numb er of a ccumulators larger than one , linea r minimum distance growt h can be obtained, but an asymp totic growth rate coe f ficient for the minimum distan ce was o nly conjectured . This was also shown in [10] for the s pecial cas e of tw o accumu lators, but again an asy mptotic growt h rate co efficient was not g i ven. Lower bo unds on the average minimum distan ce growt h for finite block leng ths are gi ven in Fig. 1 for dif ferent code rates R a nd two (RAA) and three a ccumulators (RAAA) along with the GVB. The bounds are compu ted such that half of the co des in each ensemb le h av e a minimum distance of at lea st d min . W e observe that, for rate R = 1 / 2 , the growth rate coefficient of the RAA code is much smaller 3 than the GVB; h owe ver , both the RAAA code an d the R AA c ode punc tured from a rate 1 / 3 mother c ode (R3-AAp) hav e a minimum distance growth rate coefficient very close to the GVB. Also, it can b e seen that, for code rates of R = 1 / 3 an d R = 1 / 4 , the minimum distance gro wth rate coefficient is c loser to the GVB tha n in the R = 1 / 2 cas e. For the rate R = 1 / 3 and R = 1 / 4 RAAA cod e en sembles the minimum distance g rowt h rate coefficients c oincide with the GVB an d are therefore no t shown in Fig. 1. In contrast to the results g i ven in Fig. 1, we are interes ted in the asymptotic behavior of the minimum distance for RMA codes with block len gths ten ding to infinity . In particular , an asymptotically ”go od” code ense mble has the property that, for large bloc k lengths, the minimum distanc e grows linearly with block length, wh ich cann ot be reliably d etermined from the finite length analysis shown in Fig. 1. 200 400 600 800 1000 0 50 100 150 200 250 block length d min GVB R=1/4 RAA R=1/4 GVB R=1/3 RAA R=1/3 GVB R=1/2 RAAA R=1/2 R3−AAp R=1/2 RAA R=1/2 Fig. 1. Lowe r bounds on the av erage minimum distance growth rate coefficient for finite block l engths for RAA and RAAA code ensembles along with the GVB. In the following, we extend previous res ults in the literature [8], [9], [10] a nd pres ent an analysis tha t fully characterizes the asymptotic minimum distance behavior of RA an d RMA co de ensemble s. The main result of the pa per is that, for RAA code s and rates equal to 1 / 3 or sma ller , these code ensembles exhibit linear distance growth with bloc k length, where the gap to the GVB can be made arbitrarily small by increasing the numb er of a ccumulators beyond two. In a ddition, we consider random punc turing a t the output of the last a ccumulator . W e s how that in this case the resulting ensemble o f RAA code s achieves 4 linear distanc e growt h close to the GVB for any code rate smaller than one. The p aper is organized as follo ws. In Section II we conside r the ensemble average weigh t sp ectrum of RA code s a nd s how that the minimum distan ce grows as a fractiona l power of the block le ngth. Section III addres ses the asy mptotic minimum d istance analysis of a double se rially conc atenated RAA code. This c oncep t is extended to multiple s erial conc atenation with more than two a ccumulators in Section IV . Finally , random puncturing and its effect on minimum distanc e is discusse d in Section V , and some concluding remarks are giv en in Section VI. I I . E N S E M B L E A V E R AG E W E I G H T S P E C T RU M F O R R E P E A T - A C C U M U L A T E C O D E S In this s ection we b riefly addres s the minimum distance of RA cod es a nd show tha t these cod es cann ot achieve linea r distance gro wth with block length, i.e., they are not as ymptotically good co de en sembles. Related res ults have already be en establishe d in [9], [10], [11], and [12], where lo wer a nd upper bounds on minimum distan ce for more ge neral s erially conc atenated code s hav e been d eri ved. W e restate these results for tutorial reason s and since we will us e the m in Se ction III , where we analyze the RAA co de ensemble. The RA encode r is shown in Fig. 2. T he binary inp ut seq uence u ha s length K an d Hamming weight π 1 weight q w weight w u v weight d A 1 R Fig. 2. Repeat-accumulate encoder . w , and R den otes the repetition code of rate R = 1 /q , whic h leads to a co dew ord o f weight q w an d length N = q K . The subsequ ent interleav er π 1 permutes the symbo ls of the codeword. W e cons ider the ensemble of all interleavers by us ing the uniform interleaver approac h [6], whe re each p ossible interleaver realization ha s probability 1 / N ! . Th e permuted outpu t sequ ence is applied to the recursive con vol utional code A 1 with ge nerator po lynomial g ( D ) = 1 / (1 + D ) (acc umulator), leading to a n output sequen ce v of weight d . W e will characterize RA code e nsembles b y their input-output w eight enu merating function (IO WEF) A d,w , wh ich is the n umber of co dewords having weigh t d that resu lt from inpu t se quenc es of weigh t w . Let E ( A d,w ) deno te the expected value o f the IO WEF using the uniform interleaver ap proach. 5 W e also define the ensemble-average weight enumerating function (WEF) E ( A d ) , the exp ected number of cod ew ords of weight d , as E ( A d ) , K X w =1 E ( A d,w ) , (1) and s imilarly the ensemble-average c umulativ e WEF E ( A d ≤ δ ) specifying the expe cted number of code- words in the ensemble with we ight not excee ding δ , E ( A d ≤ δ ) , δ X d =1 E ( A d ) . (2) For a given co de, the minimum distanc e d RA min is defi ned as the smallest value of δ for which A d ≤ δ is nonzero. Theorem 1 . In the ensemb le o f RA code s with block length N → ∞ and q ≥ 3 , almost all code s have minimum distanc e d RA min lower boun ded by the ineq uality d RA min > N q − 2 q − ǫ , where ǫ is any fixe d positive value. Pr o of: T he conditional probability that a weigh t d codew ord is obtained at the output of the accumulator for a giv en input weight w can be expresse d as [2], [6] Pr( d | w ) =                              1 , for w = 0 , d = 0 , 0 , for w = 0 , d ≥ 1 , 0 , for w ≥ 1 , d = 0 ,  d − 1 ⌈ q w 2 ⌉ − 1  q K − d ⌊ q w 2 ⌋   q K q w  , for w ≥ 1 , d ≥ 1 . (3) Since we are only interested in code words with nonze ro weight, in the follo wing we will only consider the ca se where w ≥ 1 . Note tha t from (3) we obtain the con straints l q w 2 m ≤ d and j q w 2 k ≤ q K − d. (4) The total nu mber of inpu t sequ ences h aving w eight w is  K w  . The n E ( A d,w ) is g i ven as E ( A d,w ) =  K w  Pr( d | w ) . (5) 6 By us ing the fact that N ℓ > ℓ − 1 Y λ =0 ( N − λ ) ≥ N ℓ ϕ N ( ℓ ) with ϕ λ ( ℓ ) , exp  ℓ ( ℓ − 1) 2 λ  , (6) we can s how that  N ℓ  ℓ 1 ϕ N ( ℓ ) ≤  N ℓ  ≤  N ℓ  ℓ ϕ ℓ ( ℓ ) . (7) Thus we c an uppe r bound (5) as E ( A d,w ) ≤ N w d ⌈ q w / 2 ⌉− 1 N ⌊ q w / 2 ⌋ q w N q w 2 q w l q w 2 m ϕ N ( w ) . (8) The cumu lati ve WEF is given by E ( A d ≤ δ ) = K X w =1 δ X d =1 E ( A d,w ) , (9) and by u sing (8) the sum over d in (9) can be upper bou nded as δ X d =1 E ( A d,w ) ≤ δ X d =1 N w d ⌈ q w / 2 ⌉− 1 N ⌊ q w / 2 ⌋ q w N q w 2 q w l q w 2 m ϕ N ( w ) < δ ⌈ q w / 2 ⌉ N q w − w −⌊ q w / 2 ⌋  2 q q  w l q w 2 m ϕ N ( w ) , (10) where w ≥ 1 , d ≥ 1 . Now ch oosing δ = N q − 2 q − ǫ , where ǫ is any fi xed positiv e value, we o btain δ X d =1 E ( A d,w ) < N − q w ǫ/ 2  2 q q  w  e ( w − 1) / (2 N )  w  ( q w ) 1 /w  w < N − q w ǫ/ 2  2 q q  w  e 1 / (2 q )  w e q w /e = η w , (11) where η = N − q ǫ/ 2 2 q q e 1 / (2 q ) e q /e , (12) and we have employed the inequa lities max 1 ≤ w ≤ K e ( w − 1) / (2 N ) < e 1 / (2 q ) and ( q w ) 1 /w ≤ e q /e . Next we cho ose a n N 0 = N 0 ( ǫ ) s uch that η 0 = N − q ǫ/ 2 0 2 q q e 1 / 2 q e q /e < 1 2 , (13) and then, for N ≥ N 0 , by c ombining (9), (11), and (12) we obtain E ( A d ≤ δ ) < ∞ X w =1 η w = η 1 − η < 1 . (14) From (14) we c onclude that there exists codes in the ensemble with b lock length N > N 0 whose minimum distan ce satisfies d RA min > N q − 2 q − ǫ . Since η / (1 − η ) → 0 as N → ∞ , the fraction of c odes in the en semble with d RA min ≤ N q − 2 q − ǫ goes to zero, which proves the the orem. 7 In the case q = 2 the above b ound does not hold. In p articular , using (5) we can show that E ( A d =1 ,w =1 ) = 1 , i.e., on average e ach RA code has one codeword of weight one generated b y a weight on e input sequ ence. I I I . E N S E M B L E A V E R AG E W E I G H T S P E C T R U M F O R R E P E A T - AC C U M U L A T E - AC C U M U L A T E C O D E S In this and the following s ection, we extend the encoder of Fig. 2 and conside r a serial conca tenation of M a ccumulators A ℓ with generator polynomials 1 / (1 + D ) se parated by interleavers π ℓ , 1 ≤ ℓ ≤ M , as s hown in F ig. 3. In particular , in this sec tion we focus on M = 2 and, base d on an average we ight u weight w R π 1 A 1 weight d 1 π 2 A 2 weight d 2 weight q w π M weight d , d M A M v Fig. 3. Repeat multiple accumulate encoder . enumerator a nalysis, show that the res ulting rep eat double a ccumulate (RAA) cod e en sembles for q ≥ 3 are as ymptotically goo d, i.e., as the block leng th N tends to infi nity , almost all c odes in the en semble exhibit linear distance growth with block length. Analogous to (3), the conditional probability that a weigh t d 1 codeword is o btained a t the output of the first accumu lator and a weigh t d codewor d at the output of the sec ond accumula tor for a g i ven input weight w is giv en as Pr( d, d 1 | w ) = Pr( d 1 | w ) Pr( d | d 1 ) =  d 1 − 1 ⌈ q w 2 ⌉ − 1  q K − d 1 ⌊ q w 2 ⌋   q K q w   d − 1 ⌈ d 1 2 ⌉ − 1  q K − d ⌊ d 1 2 ⌋   q K d 1  (15) where w , d 1 , and d must satisfy the c onstraints l q w 2 m ≤ d 1 , j q w 2 k ≤ q K − d 1 ,  d 1 2  ≤ d, and  d 1 2  ≤ q K − d. (16) After so me straightforward manipulations and, recalling that N = q K , (15) can be re written as Pr( d, d 1 | w ) =  N − q w d 1 − ⌈ q w 2 ⌉  q w ⌈ q w 2 ⌉  d 1 ⌈ d 1 2 ⌉  N − d 1 d − ⌈ d 1 2 ⌉   N d 1   N d  ·  q w 2   d 1 2  d 1 d . (17) 8 W e ca n now write the ens emble av erage conditional IOW EF as E ( A d,d 1 ,w ) =  K w  Pr( d, d 1 | w ) . (18) An up per bound on the ens emble-average WEF is giv en b y E ( A d ) = K X w =1 N X d 1 =1 E ( A d,d 1 ,w ) ≤ K N max 1 ≤ w ≤ K max 1 ≤ d 1 ≤ N E ( A d,d 1 ,w ) , (19) and a lower bound is giv en by E ( A d ) ≥ max 1 ≤ w ≤ K max 1 ≤ d 1 ≤ N E ( A d,d 1 ,w ) . (20) In a similar way , the ens emble-average cumulativ e WEF E ( A d ≤ δ ) can b e uppe r bounded by E ( A d ≤ δ ) ≤ δ K N max 1 ≤ d ≤ δ max 1 ≤ w ≤ K max 1 ≤ d 1 ≤ N E ( A d,d 1 ,w ) . (21) In the following, our g oal is to show that the ensemble-average cu mulati ve WEF tend s to z ero as N → ∞ for all δ < ˆ ρ min N , where ˆ ρ min is a lower bound on the as ymptotic minimum distance growth rate coefficient of the ens emble. W e do this by showing that N 3 E ( A d,d 1 ,w ) → 0 for all values of d , 1 ≤ d < ˆ ρ min N . T o this end, we write the weights w , d 1 , and d as w = α N a , d 1 = β N b , d = ρ N c , (22) where 0 ≤ a ≤ b ≤ c ≤ 1 , and α , β , ρ a re pos iti ve cons tants, and con dition (16) must be satisfie d. W e now d i vide the problem into the following two case s: 1) At leas t on e of the weights w , d 1 , a nd d is of order o ( N ) , so at lea st one of the constants a , b , and c is less tha n 1. 2) All the weights w , d 1 , and d can b e expres sed as fractions of the b lock length N , so a = b = c = 1 . Lemma 1. In the ensemble of RAA codes with block leng th N and q ≥ 3 , in the c ase wher e at least one of the weights w , d 1 , a nd d is of o r der o ( N ) , N 3 E ( A d,d 1 ,w ) → 0 a s N → ∞ for all values of d , 1 ≤ d < N / 2 . The proof of Lemma 1 ca n be found in Appendix A. From Lemma 1 we conclude that the contributi on of the first c ase to the cumu lati ve WEF tends to z ero as the bloc k length N tend to infinity . Thus it is sufficient to only co nsider weights that a re of the same orde r as the block length N . 9 W e now con sider the s econd case when all the weights c an be expressed a s fractions o f the block length N . Using Stirling’ s a pproximation, we have that  N d  ≈ e N · H ( d/ N ) , where H ( x ) = − x ln( x ) − (1 − x ) ln(1 − x ) is the binary entropy function. Ap plying this to (18), for N → ∞ we obta in E ( A d,d 1 ,w ) = exp ( f ( α, β , ρ ) N + o ( N )) , (23) where α = w /K = q w / N , β = d 1 / N , an d ρ = d/ N are n ormalized weights, and the function f ( α, β , ρ ) = H ( α ) q − H ( β ) − H ( ρ ) + H  β − α/ 2 1 − α  (1 − α ) + α ln 2 + H  ρ − β / 2 1 − β  (1 − β ) + β ln 2 . (24) Since the minimum distance of a ny linea r c ode can not be grea ter tha n N / 2 , we cons ider a fixed ρ ≤ 1 / 2 , and the function f ( α, β , ρ ) in (24) is defi ned o n a region G ⊂ I R 2 with bo undaries (a) ( α = 0 , 0 ≤ β ≤ 2 ρ ) , (b) (0 < α ≤ min(1 , 4 ρ ) , β = α/ 2) , (c) (0 < α ≤ min(2 − 4 ρ, 4 ρ ) , β = 2 ρ ) , (d) (2 (1 − 2 ρ ) ≤ α ≤ 1 , β = 1 − α/ 2) if ρ > 0 . 25 , (25) where the boundaries follow from the con straints in (16). Note that bounda ry (a) shou ld be excluded from region G since we exc lude the c ase w = 0 . Howev er , for the sa ke o f clarity , we call it a boun dary of G since all points ( α > 0 , β > 0) arbitrarily close to bou ndary (a) in G must be included. An example of G is shown for ρ < 0 . 25 in Fig. 4. Now let ˜ f ρ ( α, β ) , f ( α, β , ρ ) b e a func tion of α and β for a α β (b): (0 ≤ α ≤ 4 ρ, β = α / 2) (a): ( α = 0 , 0 ≤ β ≤ 2 ρ ) G (c): (0 ≤ α ≤ 4 ρ, β = 2 ρ ) 4 ρ 2 ρ Fig. 4. Properties of t he function f ( α, β , ρ ) : region G for ρ < 0 . 25 . 10 fixed ρ < 1 / 2 . From (24) it c an b e se en that the v alue of the function ˜ f ρ (0 , 0) is z ero for a ny given ρ . A sufficient con dition for linear distanc e growt h c an be stated as follows: if ˜ f ρ ( α, β ) is strictly negativ e for all 0 < ρ < ˆ ρ min and all α , 0 < α ≤ 2 min( β , 1 − β ) , a nd β , 0 < β ≤ 2 min( ρ, 1 − ρ ) , the n almos t all code s in the e nsemble have minimum distance d min = ρ min N ≥ ˆ ρ min N . Thus, to show tha t ˆ ρ min is a lower bo und on the a symptotic minimum distance g rowt h rate c oefficient o f the ens emble the ma ximum of ˜ f ρ ( α, β ) for a ll 0 < ρ < ˆ ρ min must be negati ve in G . W e now address the maximization of f ( α, β , ρ ) over α and β , where, in principle, a maximum of the function ˜ f ρ ( α, β ) can occu r inside the region G or on boundaries (b), (c), o r (d). Howev er , we will show below that the maximum only occ urs inside the region G . Lemma 2. F or any given ρ ≤ 1 / 2 , the stationary points of the func tion ˜ f ρ ( α, β ) satisfy the following system of eq uations:  β − α/ 2 1 − α   1 − β − α/ 2 1 − α  =  α 1 − α  2 q , (26) 4  ρ − β / 2 1 − β   1 − ρ − β / 2 1 − β  =  1 − β β β − α/ 2 1 − β − α/ 2  2 . (27) Pr o of: The partial de ri v ati ves o f the function ˜ f ρ ( α, β ) are giv en as ∂ ˜ f ∂ α = − 1 q ln  α 1 − α  + 1 2 ln  β − α/ 2 1 − α  + 1 2 ln  1 − β − α/ 2 1 − α  + ln 2 , (28) ∂ ˜ f ∂ β = ln  β 1 − β  − ln  β − α/ 2 1 − β − α/ 2  + 1 2 ln  ρ − β / 2 1 − β  + 1 2 ln  1 − ρ − β / 2 1 − β  + ln 2 . (29) At the stationary points o f ˜ f ρ ( α, β ) , we have ∂ ˜ f /∂ α = 0 and ∂ ˜ f /∂ β = 0 . By setting (28 ) to zero and letting x , β − α/ 2 1 − α , we obtain 4 x (1 − x ) =  α 1 − α  2 q , which is ide ntical to (26). Like wise, by se tting (29) to zero with y , ρ − β / 2 1 − β we obtain 4 y 2 − 4 y +  1 − β β  2  β − α/ 2 1 − β − α/ 2  2 = 0 , which leads to (27). In order to find the stationary points of ˜ f ρ ( α, β ) , we must solve (26) and (27 ), where ρ is expressed as a function of α a nd β . W e first con sider (27), wh ich can b e viewed as a q uadratic equ ation in ρ with variables α an d β . It follo ws that, if β > 1 / 2 , (27) has no real-valued z eros. On the other ha nd, if β ≤ 1 / 2 , then (27) contains two real-valued zeros: ρ 1 ( α, β ) ≤ 1 / 2 and ρ 2 ( α, β ) ≥ 1 / 2 . Since we on ly 11 consider the c ase ρ ≤ 1 / 2 , the solution of the qua dratic equation (27) in ρ then y ields ρ = ρ 1 ( α, β ) = 1 2 − 1 − β 2 s 1 −  1 − β β β − α/ 2 1 − β − α/ 2  2 , (30) which relates the quan tities α , β , and ρ . Now c onsider (26) as a quadratic eq uation in β w ith variable α . For α > 1 / 2 there are no real-valued zeros, b ut for α ≤ 1 / 2 (26) has two real-valued zeros: β 1 ( α ) ≤ 1 / 2 and β 2 ( α ) ≥ 1 / 2 . Since ρ 1 ( α, β ) is real-valued only if β ≤ 1 / 2 , the solution of the qua dratic e quation (26) in β is giv en by β = β 1 ( α ) = 1 2 − 1 − α 2 s 1 −  α 1 − α  2 q . (31) A p air ( α, β ) max imizing 1 ˜ f ρ ( α, β ) c an be comp uted from (30 ) an d (31) for a g i ven ρ . The re is a lower limit ρ 0 for w hich the system of e quations has a solution. If ρ < ρ 0 , (30) and (31) have n o solution and ˜ f ρ ( α, β ) has no stationary points within G . For ρ > ρ 0 , (30) a nd (31) yield two solutions, where one s olution correspon ds to a ma ximum and the other to a saddle point. Finally , there is only a single stationary point for ρ = ρ 0 . Fig. 5 shows the ma xima ˜ f max ρ inside the region G for dif ferent values of q , 0.1 0.15 0.2 0.25 0.3 −0.02 0 0.02 0.04 0.06 0.08 0.1 ρ f max ρ q=3 q=4 q=5 q=6 Fig. 5. Maxima ˜ f max ρ inside the region G for the R AA code ensemble with rates R = 1 / 3 , 1 / 4 , 1 / 5 , 1 / 6 . where we obse rve a zero cross ing at ρ = ˆ ρ min , 0 < ˆ ρ min ≤ 0 . 5 . T o s how that ˆ ρ min is indeed the asymp totic 1 The structure of (30) and (31) suggest a slightly dif ferent but equi v alent approach, w here α is used as the free parameter and β and ρ are determined from (30) and ( 31). 12 minimum distance growth rate co efficient, we n ow address the behavior of the function ˜ f ρ ( α, β ) at the bounda ries o f G . Lemma 3 . The function ˜ f ρ ( α, β ) canno t hav e a ma ximum on boundaries (a), (b), and (d) but ca n have a max imum on boun dary (c). Pr o of: W e introduce the function ϕ ρ,β ( α ) , f ( α, β , ρ ) for fixed β and ρ , wh ich is defined on a line parallel to the α -axis (se e Fig. 6 for an illustration), and conside r the followi ng three c ases : 1) For 0 < β ≤ m in (0 . 5 , 2 ρ ) , we fix the no rmalized we ight β and allow α to vary between 0 and 2 β . In this c ase, ϕ ρ,β ( α ) is defi ned o n a line from bounda ry (a) to (b) in (25). 2) For 1 / 2 < β ≤ 2 ρ , we fix β an d allow α to vary between 0 and 2(1 − β ) . In this c ase, the func tion ϕ ρ,β ( α ) is define d on a line from bou ndary (a) to (d). 3) For β = 2 ρ , the n ormalized input weight α varies be tween 0 and m in(2 − 4 ρ, 4 ρ ) , i.e., ϕ ρ,β ( α ) is defined on b oundary (c). The d eri vati ve dϕ ρ,β ( α ) /dα is given by (28) in all three ca ses. Note that, at the stationa ry points, α an d β are related b y (28), independe nt of ρ : if β ≤ 1 / 2 , then α can be ob tained by solving (31), a nd if β > 1 / 2 , we ca n find α by s olving β = 1 2 + 1 − α 2 s 1 −  α 1 − α  2 q . The sec ond deriv a ti ve d 2 ϕ ρ,β ( α ) /dα 2 is given b y d 2 ϕ ρ,β ( α ) dα 2 = − 1 q 1 α (1 − α ) + 1 2  1 − 2 x x (1 − x )  2 β − 1 (1 − α ) 2 , (32) where x , β − α/ 2 1 − α and 0 ≤ x < 1 / 2 . At a stationary point with α = α 0 , the corres ponding β can b e obtained from (31 ). For such a pair ( α 0 , β ) , we then obtain from (32) that d 2 ϕ ρ,β ( α ) dα 2     α = α 0 = − 1 q α 0 (1 − α 0 ) − (1 − 2 β ) 2 (1 − α 0 ) ( β − α 0 / 2) (1 − β − α 0 / 2) < 0 . It follows tha t the s tationary point on a con stant β line in the ( α, β ) p lane corresp onds to a max imum for each ( α 0 , β ) pair sa tisfying (31). Conse quently , for c ases 1) an d 2), the max imum of the func tion ˜ f ρ ( α, β ) ca nnot be on the bound aries (a), (b), an d (d). For the same rea son, in case 3), the maximum of the line ˜ f ρ ( α, β )   β =2 ρ is located on the bou ndary (c). Lemma 4. F or any ρ < 1 / 2 and q ≥ 3 the function f ( α, β , ρ ) is ne gative on the bound aries (a) and (b) of the r e gion G excep t for the po int ( α, β ) = (0 , 0) , where f (0 , 0 , ρ ) = 0 for any ρ < 1 / 2 . 13 The proof of Lemma 4 can be found in Appen dix B. A numerical ana lysis shows that the max imum value on b oundary (c) is always less than the ma ximum inside the region G , if it exists, or s trictly nega ti ve if there is no stationary po int inside G . And since the function f ( α, β , ρ ) is always nega ti ve on the bound aries (a) and (b), except for the point ( α, β ) = (0 , 0) , we ne ed not con sider the values on the bounda ry of the region G in (25), a nd we co nclude that, for all ρ < ˆ ρ min , the func tion ˜ f ρ ( α, β ) is negative. Thus, we ob tain from Lemmas 2, 3, and 4 that ˆ ρ min is a lower b ound on the asymptotic minimum distance growth rate coefficient of the c ode ensemb le. W e summarize our fi ndings in the followi ng theo rem. Theorem 2. In the e nsemble of RAA codes of rate R ≤ 1 / 3 and block len gth N → ∞ , almost all codes h ave minimum distance d min gr o wing linearly with N . A lower bound on the asymptotic minimum distance gr owth rate coefficient ρ RAA min = d RAA min / N of the e nsemble can be o btained by solving the sy stem of equ ations (30) and (31), i.e., by finding the max imum of the func tion f ( α, β , ρ ) . T o illustrate the be havior of the function ˜ f ρ ( α, β ) , Fig. 6 shows two examp les of contour plots o f ˜ f ρ ( α, β ) for the RAA code ens emble with q = 3 a nd d if ferent values of ρ . For ρ = 0 . 1 , d isplayed β α 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 −0.18 −0.16 −0.14 −0.12 −0.1 −0.08 −0.06 −0.04 −0.02 (a) ρ = 0 . 1 β α 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.05 0.1 0.15 0.2 0.25 0.3 0.35 −0.2 −0.15 −0.1 −0.05 0 (b) ρ = 0 . 2 Fig. 6. Contour plots of ˜ f ρ ( α, β ) = f ( α, β , ρ ) for a fixed ρ over the region G . Maximum values can only be located along the dashed line wit hin G . in Fig. 6(a), there is no stationary point inside the re gion G . The function ˜ f ρ ( α, β ) is decreasing monotonically from the origin towards b oundary (c), located at the top of Fig. 6(a). By c ontrast, for ρ = 0 . 2 , Fig. 6(b) clearly shows a maximum ins ide G . No te that the pa irs ( α 0 , β ) satisfying (31) 14 correspond to the dashed lines in Fig. 6, which indicate the poss ible locations o f stationa ry points in the ( α, β ) plane . The values of ˆ ρ min are listed in T able I a long with the corresp onding values of the GVB, the estimated minimum distance gro wth rate coefficient ˜ ρ min based on a finite block leng th analysis an alogous to the one presented in Fig. 1 , and the values ρ 0 . W e se e that, es pecially for lower code rates, the asymp totic minimum distan ce growt h rate coefficient of the RAA en semble is close to the GVB. Also, the resu lts of the finite length analysis match the asymp totic resu lts quite well. T ABL E I A S Y M P T OT I C M I N I M U M D I S TA N C E G R OW T H R A T E C O E FFI C I E N T L OW E R B O U N D ˆ ρ MI N , E S T I M A T E D M I N I M U M D I S TA N C E G R O W T H R AT E C O E FFI C I E N T ˜ ρ MI N F R O M A FI N I T E L E N G T H A NA LY S I S , A N D T H E C O R R E S P O N D I N G V A L U E S O F T H E G V B F O R T H E R A A E N S E M B L E W I T H D I FF E R E N T C O D E R A T E S . T H E V A L U E S ρ 0 D E N OT E T H E S M A L L E S T ρ F O R W H I C H A S O L U T I O N O F T H E M A X I M I Z AT I O N P RO B L E M G I V E N B Y (26) A N D (27) C A N B E O B TA I N E D . q ˆ ρ min ˜ ρ min GVB ρ 0 3 0.1323 0.1339 0.1740 0.1225 4 0.1911 0.1935 0.2145 0.1742 5 0.2286 0.2312 0.2430 0.2075 6 0.2549 0.2570 0.2644 0.2309 For q = 2 , t here exists no such lower bound on the asymptotic mi nimum distance growth rate coefficient. In this cas e, for any ¯ ρ > 0 the cumulative WEF of the RAA cod e ensemble can be lower bounde d by E ( A d ≤ ¯ ρN ) = K X w =1 N X d 1 =1 ¯ ρN X d =1 E ( A d,d 1 ,w ) ≥ ¯ ρN X d =1 E ( A d,d 1 =1 ,w =1 ) = ¯ ρN X d =1 1 N = ¯ ρ. (33) Even though we expect the majority of codes in the ens emble to have a minimum distance that grows linearly with bloc k length [9], for any fixed ¯ ρ > 0 , the re is a non v anishing fraction o f c odes in the ensemble with minimum distan ce d min < ¯ ρN . Th us, for the RAA ensemble with q = 2 , we c annot giv e a lower bound on the asymptotic minimum distance growth rate coe f ficient. 15 I V . E N S E M B L E A V E R A G E W E I G H T S P E C T RU M F O R R E P E A T M U LT I P L E A C C U M U L A T E C O D E S W e now generalize the results of the previous section to RMA codes with M > 2 , i.e ., with more than two accumulators (see Fig. 3) 2 . In this case, the conditional prob ability of the weight vector d = [ d 1 , d 2 , . . . , d M ] for a given inpu t weight can be written as Pr( d | w ) = Pr( d 1 | w ) · M Y ℓ =2  d ℓ − 1 ⌈ d ℓ − 1 2 ⌉ − 1  q K − d ℓ ⌊ d ℓ − 1 2 ⌋   q K d ℓ − 1  , (34) where Pr( d 1 | w ) is defined in (3 ). The e nsemble average IO WEF is the n giv en by E ( A d ,w ) =  K w  Pr( d | w ) = exp ( f ( γ ) N + o ( N )) , (35) where Stirling’ s app roximation for large N h as a gain been employed. The vector γ contains normalized weights and is given by γ = [ β 0 , β 1 , β 2 , . . . , β M ] =  α = w K , d 1 N , d 2 N , . . . , ρ = d M N  , where β 0 , α a nd β M , ρ . The function f ( γ ) in (35) ca n now be written as f ( γ ) = H ( α ) q − M − 1 X ℓ =1 H ( β ℓ ) + M X ℓ =1 H  β ℓ − β ℓ − 1 / 2 1 − β ℓ − 1  (1 − β ℓ − 1 ) + ln 2 M − 1 X ℓ =0 β ℓ , (36) which represen ts a ge neralization o f the func tion defined in (24) to more than three normalized weight terms. Analogou s to the deriv a tion for the RAA cas e in Section III, f ( γ ) must now be maximized over α, β 1 , . . . , β M − 1 . Here, t he same ar g uments for t he existence of stationary points on the boundary or inside an M -dimension al region G M can b e made, analogo us to the RAA case , where again only the maximum inside G M must b e conside red in the maximization problem. Thus, the M + 1 tuple ( α, β 1 , . . . , β M − 1 , ρ ) maximizing f ( γ ) c an be expres sed by the following set of rec ursiv e e quations: β 1 = 1 2 − 1 − α 2 s 1 −  α 1 − α  2 q and (37) β ℓ +1 = 1 2 − 1 − β ℓ 2 s 1 −  1 − β ℓ β ℓ · β ℓ − β ℓ − 1 / 2 1 − β ℓ − β ℓ − 1 / 2  2 , (38) 1 ≤ ℓ ≤ M − 1 , for any α suc h that 0 < α ≤ 1 / 2 . The deriv ation of this s et of equa tions follo ws from the deri v ation of (30) and (31) in a straightforw ard way: (37) is equiv ale nt to (31) with β replac ed b y β 1 , and (38) is a generaliza tion of (30) with ρ rep laced by β ℓ +1 and β by β ℓ . 2 In contrast to the M = 2 case, for M > 2 we are able to deriv e a lower bound on the asymptotic minimum distance gro wth rate coefficient for the rate R = 1 / 2 code ensemble. 16 As an example, in Fig. 7 the values ˆ ρ min for the asymp totic minimum distance growth rate c oefficient are shown for M = 2 and M = 3 cod e en sembles for q = 2 , 3 , 4 , 5 , and 6 and compared to the GVB. The values of ˆ ρ min , a long with the estimated v alues ˜ ρ min obtained from a finite-length analysis and the 0.1 0.2 0.3 0.4 0.5 0.6 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 R ρ min ^ GVB R q AA R q AAA q = 2 q = 3 q = 4 q = 5 q = 6 Fig. 7. GVB and the corresponding asymptotic minimum distance growth rate coef ficient lower bound ˆ ρ min for RAA and RAAA code ensembles wit h rates R = 1 / 2 , 1 / 3 , 1 / 4 , 1 / 5 , and 1 / 6 . GVB, are listed in T able II. W e observe that the res ulting as ymptotic minimum distan ce growth rate T ABL E II A S Y M P T OT I C M I N I M U M D I S TA N C E G R OW T H R A T E C O E FFI C I E N T L OW E R B O U N D ˆ ρ MI N , E S T I M A T E D M I N I M U M D I S TA N C E G R O W T H R AT E C O E FFI C I E N T ˜ ρ MI N F R O M A FI N I T E L E N G T H A NA LY S I S , A N D T H E C O R R E S P O N D I N G V A L U E S O F T H E G V B F O R T H E R A A A E N S E M B L E W I T H D I FF E R E N T C O D E R A T E S . q ˆ ρ min ˜ ρ min GVB 2 0.1034 0.1109 0.1100 3 0.1731 0.1739 0.1740 4 0.2143 0.2150 0.2145 5 0.2428 0.2400 0.2430 6 0.2643 0.2627 0.2644 17 coefficients of the RAAA code ens emble for q ≥ 3 essen tially ac hieve the GVB, which is consistent with the resu lts obtained in [8 ] for finite block leng ths and a n umber of a ccumulators tending to infinity . Analogous to Theorem 2, we now s tate the following theorem. Theorem 3. In the ens emble of R MA codes w ith M accumulators, M ≥ 3 , of rate R ≤ 1 / 2 and block length N → ∞ , almos t all co des have minimum distance d min gr o wing linearly with N . A lowe r boun d on the asymptotic minimum distance gr owth rate coefficient ρ RMA min = d RMA min / N of the ense mble ca n be obtained by s olving the sy stem of eq uations (37) and (38) , i.e., by finding the maximum of the function f ( γ ) . V . R E P E A T M U L T I P L E A C C U M U L A T E C O D E S W I T H R A N D O M P U N C T U R I N G The rate o f RMA cod e ense mbles is determined by the rate of the outer repetition cod e. Thus it is not possible to obta in rates higher than 1/2 withou t puncturing. This motiv ates u s to employ random puncturing at the outpu t of the inner accumu lator in con nection with a lo wer- rate RMA mother c ode for the pu rpose of ach ieving higher rate RMA cod e ensembles . Let N ′ be the numbe r of code symbo ls after pu ncturing, d ′ the correspon ding codeword weight, and R ′ = R · N / N ′ the code rate. W e also define the ratios η = N ′ / N , the normalized block length, and ρ ′ = d ′ / N ′ , the normalized output weight, after puncturing. The co nditional proba bility of a weight- d ′ sequen ce a fter puncturing is g i ven by the hypergeometric distribution Pr N ′ ( d ′ | d ) =  d d ′  N − d N ′ − d ′   N N ′  , which for lar ge N ca n be expres sed a s Pr N ′ ( d ′ | d ) = exp  N  H  η ρ ′ ρ  ρ + H  η (1 − ρ ′ ) 1 − ρ  (1 − ρ ) − H ( η )  + o ( N )  . (39) Considering the g eneral cas e of repeat multiple ac cumulate codes , the e nsemble average IO WEF can now be obtained from (35) as follows: E ( A d ,d ′ ,w ) =  K w  Pr( d | w ) P r N ′ ( d ′ | d ) = exp  F ( γ , ρ ′ , η ) N + O (ln N )  , where Pr( d | w ) is de fined in (34), F ( γ , ρ ′ , η ) , f ( γ ) + ϕ ( ρ ′ , ρ, η ) , and ϕ ( ρ ′ , ρ, η ) = H  η ρ ′ ρ  ρ + H  η (1 − ρ ′ ) 1 − ρ  (1 − ρ ) − H ( η ) . (40) 18 Follo wing the approa ch us ed for the RAA e nsemble in S ection III, the max imization of the fun ction F ( γ , ρ ′ , η ) must now be carried out over a ll elements of the vector γ , including ρ = β M . Ag ain, for the same reas ons as in the RAA ca se, we consider only stationa ry points of F ( γ , ρ ′ , η ) inside the M + 1 - dimensional region spann ed by the M + 1 tup le γ . Note that ϕ ( ρ ′ , ρ, η ) in (40) does not dep end on the variables α, β 1 , . . . , β M − 1 , since only the outpu t of the inner en coder is pun ctured. Th erefore we c an s till make use of (37) and (38) for 1 ≤ ℓ ≤ M − 1 . In add ition, we nee d to compute the deriv a ti ve ∂ F /∂ ρ , wh ich is given by ∂ F ∂ ρ = ln  ρ 2 (1 − ρ − β M − 1 / 2) (1 − ρ ) 2 ( ρ − β M − 1 / 2)  + ln  1 − ρ − η + ρ ′ ρ − ρ ′  . (41) W e then solve ∂ F /∂ ρ = 0 for ρ ′ , which yields ρ ′ = ρ ( c + 1) + η − 1 1 + c , where c = (1 − ρ ) 2 ( ρ − β M − 1 / 2) ρ 2 (1 − ρ − β M − 1 / 2) . (42) W e can now search for a maximum of F ( γ , ρ ′ , η ) by using (37 ) an d (38), for 1 ≤ ℓ ≤ M − 1 , and (42). Fig. 8 conside rs the particularly interesting RAA case and shows the lo wer bound on the asymptotic minimum distance growth rate coefficient ˆ ρ min for different mother c ode ra tes R and pun ctured c ode rates R ′ . W e obs erve that, c ompared to the unpun ctured RAA c ode ensemble considered in Section III, the asymptotic minimum distanc e g ro wth rate coe f ficients are closer to the GVB for the punctured ens embles with rates R ′ > R . Th is behavior is due to the extra randomne ss added by pu ncturing the enc oder output. W e also see that the growth rate coefficients approach the GVB as the rate increas es. W e co njecture that this is due to the fact that a s maller value of η leads to a lar ger rando m p uncturing e nsemble. In other words, a smaller N ′ results in a more ”ra ndom-like” c onstruction. T ab le III gives the lower bound on the asymptotic minimum distance growth rate co efficient ˆ ρ ′ min , along with the estimated v alues from a finite-length analys is ˜ ρ ′ min and the GVB, for a rate R = 1 / 3 mother c ode. The following theorem is a nalogous to Theorems 2 and 3. Theorem 4. Consider the ensemble of R MA co des with M accumulators, M ≥ 2 , of rate R ′ after random puncturing. If the block length N ′ → ∞ , almost all co des in the en semble have minimum distance d min gr o wing linearly with N ′ . A lowe r bo und on the a symptotic minimum dis tance gr owth rate c oefficient ρ ′ min of the ensem ble can be obtained by solving the system of equations (37 ) , (38) , and (42) , i.e., by finding the ma ximum of the function F ( γ , ρ ′ , η ) . V I . C O N C L U D I N G R E M A R K S W e hav e shown that RAA code e nsembles for cod e rates e qual to 1 / 3 or sma ller are a symptotically good in the s ense that their average minimum distance grows linea rly with block length N as N → ∞ . 19 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 0.05 0.1 0.15 0.2 R’ d min /N q=3 q=4 q=5 GVB ˆ ρ ′ min Fig. 8. GVB and the correspond ing normalized asymptotic minimum distance lower bound ˆ ρ ′ min for the randomly punctured RAA code ensemble wit h mother code rates R = 1 /q = 1 / 3 , 1 / 4 , 1 / 5 . T ABL E III A S Y M P T OT I C M I N I M U M D I S TA N C E G R OW T H R AT E C O E FFI C I E N T L O W E R B O U N D ˆ ρ ′ MI N , E S T I M A T E D M I N I M U M D I S TA N C E ˜ ρ ′ MI N F R O M A FI N I T E L E N G T H A NA LY S I S , A N D T H E C O R R E S P O N D I N G V A L U E S O F T H E G V B F O R T H E P U N C T U R E D R A A E N S E M B L E W I T H D I FF E R E N T C O D E R A T E S E M P L O Y I N G A M O T H E R C O D E O F R A T E 1 / 3 . R ˆ ρ ′ min ˜ ρ ′ min GVB 0.4 0.1242 0.1256 0.1461 0.5 0.1036 0.1039 0.1100 0.6 0.0771 0.0782 0.0794 0.7 0.0522 0.0530 0.0532 0.8 0.0306 0.0314 0.0311 0.9 0.0125 0.0133 0.0130 Moreover , we have shown that the asy mptotic growth rate co efficients approach the GVB for small co de rates. This extends the results of [8], where linear distance growt h is only shown for a n infinite numbe r of acc umulators. Th ese new results also exten d thos e in [9] and [10], where linear distance gro wth for the RA A ense mble is s hown, but a gro wth rate c oefficient is either not given or only co njectured. Similar results are also o btained for RMA co de ensembles with M ≥ 3 a nd cod e rates equ al to 1 / 2 or sma ller . Further , by introducing rand om puncturing at the output of the inn er acc umulator , we d emonstrate that 20 the resulting high rate RMA ens embles exhibit linear distanc e growth, where the asymptotic growth rate coefficient is close to the GVB if the mothe r code rate is sufficiently low . Despite the fact that the RMA c ode ensembles con sidered in this pa per are asymptotically good, the con ver g ence behavior of the se cod es may n ot be sufficient to provide an iterative decoding thresho ld close to c apacity , as can be seen from the simulation re sults presented in [8]. Howe ver , the results ob tained ma y be useful in c onstructing similar cod e en sembles b ased on simple co mponent codes with low en coding complexity , as ymptotically linear distan ce growth, and goo d con ver gence beh avior . In p articular , for the interesting class o f do uble serially c oncaten ated codes , the RAA ens emble ca n be use d as a starting point to de sign asymptotica lly goo d cod e co nstructions by replacing o ne o r more of the co nstituent encode rs w ith sma ll memory co n volutional codes , w hose co de polynomials c an be chosen to improve iterati ve dec oding conv er gence behavior . Some initial experimental resu lts in this direction for d if ferent compone nt encode rs are presented in [13], [14]. Another ap proach is to consider hybrid conc atenated coding sc hemes, where parts of the en coder are s tructurally equ i valent to RAA e ncoders. Initial results for low code rates hav e shown that thes e schemes have impr oved threshold b ehavior compared to RAA codes , while still p roviding asymptotica lly linear distance growth, albeit with a smaller growt h rate coefficient [15]. A P P E N D I X A. Pr oof of Lemma 1 W e conside r the followi ng five pa rtial cases : 1 − 2 q ≤ a = b ≤ c < 1 (43) 1 − 2 q ≤ a = b < c ≤ 1 (44) 0 ≤ a < b ≤ c < 1 (45) 0 ≤ a < b < c =1 (46) 0 ≤ a < b = c =1 . (47) From Theo rem 1 it follo ws that d 1 ≥ β N 1 − 2 /q for almos t all c odes in the ense mble and thus we do n ot need to co nsider the ca se of b < 1 − 2 q . In add ition to (7), we will us e the ine quality [16] s N 8 l ( N − l ) exp  H  l N  N  ≤  N l  ≤ s N 2 π l ( N − l ) exp  H  l N  N  , (48) 21 or the equiv alent expression  N l  = exp  H  l N  N + o ( N )  (49) to boun d binomial coefficients. (a) C ases (43) and (44): Using (7), (22), an d (48), we c an rewrite E ( A d 1 ,w ) =  K w  Pr( d 1 | w ) as E ( A d 1 ,w ) =  N q αN a  αN a exp  H  q α 2 β  β N a   2 N q αN a  qα 2 N a  N q αN a  − q αN a c N ( w ) , (50) where c N ( w ) = ϕ w ( w )  ϕ qw 2 ( q w 2 )  2 ϕ N ( q w ) = exp[ o ( N a ln N )] (51) is a second o rder term. For simplicity we ass ume  x 2  =  x 2  = x 2 , which is v alid for any ev en integer x an d approximately valid for large odd x . Th en we have from (50) that E ( A d 1 ,w ) = exp h 1 − q 2  α N a ln N + o ( N a ln N ) i . (52) Since E ( A d 1 ,w ) → 0 a s N → ∞ , it follows that lim N →∞ N 3 E ( A d,d 1 ,w ) = 0 independe ntly o f d for almos t all co des in the e nsemble. (b) Ca se (45): T o make the expre ssions mo re c ompact, we will omit second order terms from now on. Again, using (7), (22), and (48), we can write E ( A d 1 ,w ) ≈  N q αN a  αN a  2 β N b q αN a  qα 2 N a  2 N q αN a  qα 2 N a  N q αN a  − q αN a = exp h (1 − a )  1 − q 2  α + ( b − a ) q α 2  N a ln N + o ( N a ln N ) i (53) and Pr( d | d 1 ) ≈  2 ρN c β N b  β 2 N b  2 N β N b  β 2 N b  N β N b  − β N b = exp  − (1 − b ) β 2 N b ln N + o ( N b ln N )  . (54) Since b > a , it follows from (53) and (54) that E ( A d,d 1 ,w ) = exp  − (1 − b ) β 2 N b ln N + o ( N b ln N )  (55) and N 3 E ( A d,d 1 ,w ) → 0 as N → ∞ . 22 (c) C ase (46): In this c ase E ( A d 1 ,w ) is s till given by (53), but Pr( d | d 1 ) ≈  2 ρN β N b  β 2 N b  2(1 − ρ ) N β N b  β 2 N b  N β N b  − β N b = exp  β 2 N b ln(4 ρ (1 − ρ )) + o ( N b )  . (56) Since b > a , it follows from (53) and (56) that E ( A d,d 1 ,w ) = exp  β 2 N b ln(4 ρ (1 − ρ )) + o ( N b )  (57) and N 3 E ( A d,d 1 ,w ) → 0 as N → ∞ for ρ < 1 / 2 . (d) Ca se (47): In this c ase E ( A d 1 ,w ) is s till given by (53), but Pr( d | d 1 ) =  d  d 1 2   N − d  d 1 2    N d 1   d 1 2  d =  d 1  d 1 2   N − d 1  d − d 1 2    N d   d 1 2  d = exp  β ln 2 + H  ρ − β / 2 1 − β  (1 − β ) − H ( ρ ) | {z } , F ρ ( β )  N + o ( N )  . (58) The function F ρ ( β ) = β ln 2 + H  ρ − β / 2 1 − β  (1 − β ) − H ( ρ ) (59) is strictly n egati ve for all ρ < 1 / 2 an d β ≤ 2 ρ , which can be shown as follo ws . The deriv ati ve of F ρ ( β ) , ∂ F ∂ β = 1 2 ln  ρ − β / 2 1 − β  + 1 2 ln  1 − ρ − β / 2 1 − β  + ln 2 = 1 2 ln (4 x (1 − x )) , (60) is negati ve if x = ρ − β / 2 1 − β 6 = 1 / 2 an d equa ls zero if x = 1 / 2 . F ρ ( β ) is thus negative for all ρ ≤ 1 / 2 and all 0 < β ≤ 2 ρ . From (53) a nd (58) follows that E ( A d,d 1 ,w ) = exp  β ln 2 + H  ρ − β / 2 1 − β  (1 − β ) − H ( ρ )  N + o ( N )  (61) and N 3 E ( A d,d 1 ,w ) → 0 as N → ∞ for all ρ < 1 / 2 . 23 B. Pr oof of Lemma 4 (a) O n the bound ary ( α = 0 , 0 ≤ β ≤ 2 ρ ) , we have ∂ f ∂ β     α =0 = 1 2 ln  ρ − β / 2 1 − β  + 1 2 ln  1 − ρ − β / 2 1 − β  + ln 2 = 1 2 ln ( 4 x β (1 − x β )) < 0 (62) where x β = ρ − β / 2 1 − β . Since f ( α, β , ρ )   α =0 ,β =0 = 0 , we o btain f ( α, β , ρ ) < 0 on the boundary (a) for all β > 0 a nd ρ ≤ 1 / 2 . (b) On the bounda ry (0 ≤ α ≤ min(1 , 4 ρ ) , β = α/ 2) the total deriv a ti ve with res pect to α is given as d f dα     β = α/ 2 = ∂ f ∂ α     β = α/ 2 + ∂ f ∂ β     β = α/ 2 dβ dα = − 1 q ln  α 1 − α  + ln 2 + 1 2 ln  α/ 2 1 − α/ 2  + 1 2  1 2 ln  ρ − β / 2 1 − β  + 1 2 ln  1 − ρ − β / 2 1 − β  + ln 2  . (63) Note that the inequa lity 1 q ln  α 1 − α  > 1 2 ln  α/ 2 1 − α/ 2  + ln 2 (64) holds for q ≥ 3 and 0 ≤ α < 1 . The last term on the right hand s ide of (63) is equiv alent to (62). By inserting (62) an d (64) into (63) we obtain d f /dα   β = α/ 2 < 0 . Therefore, we conc lude that f ( α, β , ρ ) is negative along the b oundary (0 ≤ α ≤ min(1 , 4 ρ ) , β = α/ 2) for all 0 ≤ ρ < 0 . 5 . R E F E R E N C E S [1] J. Kliewer , K. S. Zigangirov , and D. J. Costello, Jr ., “New results on t he minimum distance of repeat-multiple-accumulate codes, ” in Pr oc. 45th Annual Allerton Confer ence on Communica tion, Contr ol, and Computing , Monticello, IL , S eptember 2007, pp. 1097–1102. [2] D. Div salar , H. Jin, and R. J. McEliece, “Coding t heorems for ’turbo-like’ codes, ” in Proc. 36th Annual Al lerton Conf. C ommun., Contr ol, Computing , Monticello, IL, Sept. 1998, pp. 201–210. [3] T . J. Richardson and R. L. Urbanke , “Efficient encoding of lo w-density parity-check codes, ” IEEE T rans. Inf. Theory , v ol. 47, no. 2, pp. 638–656 , Feb . 2001. [4] S. Benedetto, D. Divsalar , G. Montorsi, and F . Pollara, “ Analysis, design, and iterative decoding of double serially concatenated codes wi th interleave rs, ” IEEE J. Sel. Areas i n Commun. , vol. 16, no. 2, pp. 231–244 , Feb . 1998. [5] H. Jin and R. McEliece, “Coding theorems for turbo code ensembles, ” IEE E T rans. Inf. Theory , vol. 48, no. 6, pp. 1451–1 461, June 2002. [6] S. Benedetto, D. Div salar , G. Montorsi, and F . Pollara, “Serial concatenation of interl eav ed codes: performance analysis, design, and iterative decoding, ” IEEE T ra ns. Inf. Theory , vol. 44, no. 3, pp. 909–92 6, May 1998. 24 [7] H. D. P fister and P . H. Siegel, “Coding theorems for generalized repeat accumulate codes, ” in Pro c. International Symposium on I nformation Theory and Its Applications , Honolulu, HI, Nov . 2000, pp. 21–25. [8] H. D. P fister and P . H. Siegel, “The serial concatenation of rate-1 codes through uniform random interleav ers, ” IEEE T rans. Inf. Theory , vol. 49, no. 6, pp. 1425–14 38, June 2003. [9] H. D. Pfister , On the Capacity of Finite State Channels and the Analysis of Con volutional Accumulate-m Codes , Ph.D. thesis, Univ ersity of California S an Diego, 2003. [10] L. Bazzi, M. Mahdian, and D. A. Spielman, “The minimum distance of turbo-like codes, ” Submitted to IEEE T rans. Inf. Theory , http://www .mahdian.info/tc3.ps, May 2003. [11] N. K ahale and R. Urbanke, “On the minimum distance of parallel and serially concatenated codes, ” in Pr oc. IEEE Int. Symposium on Inform. Theory , Cambridge, MA, Aug. 1998, p. 31. [12] A. Perotti and S. Benedetto, “ An upper bound on the minimum distance of serially concatenated conv olutional codes, ” IEEE T ra ns. Inf. Theory , vol. 52, no. 12, pp. 5501–5 509, Dec. 2006. [13] F . V atta, J. Kliewer , C . Koller , D. J. Costello, Jr ., and K. S . Zigangirov , “On t he design of double serially concatenated codes with an outer repetition code, ” in Pr oc. 7th International ITG Confere nce on Sour ce and Channel Coding (SCC ’08) , Ulm, Germany , Jan. 2008. [14] C. Koller , J. Kliewer , K. S . Zigangirov , and D. J. Costello, Jr ., “Minimum distance bounds for multi ple-serially concatenated code ensembles, ” in Pr oc. IEEE Int. Symposium on Inform. Theory , T oronto, Canada, July 2008. [15] C. K oller , A. Grell i Amat, J. Kliewer , F . V atta, and D. J. Costello, Jr., “Hybrid concatenated codes with asymptotically good distance growth, ” in Proc. 5th International Symposium on Tu rbo Codes and Related T opics , Lausanne, Switzerland, Sept. 2008. [16] R. G. Gallager , Low-density parity-chec k codes , MIT Press, Cambridge, MA, 1963.