Lossy Compression of Decimated Gaussian Random Walks

Lossy Compr ession of Decimated Gaussian Random W alks Georgia Murray , Alon Kipnis, and Andrea J. Goldsmith Department of Electrical Engineering Stanford Univ ersity , Stanford, CA, USA Abstract — W e consider the pr oblem of estimating a Gaussian random walk fr om a lossy compr ession of its decimated version. Hence, the encoder operates on the decimated random walk, and the decoder estimates the original random walk from its encoded version under a mean squared error (MSE) criterion. It is well-known that the minimal distortion in this problem is attained by an estimate-and-compress (EC) source coding strategy , in which the encoder first estimates the original random walk and then compresses this estimate subject to the bit constraint. In this work, we derive a closed-form expres- sion f or this minimal distortion as a function of the bitrate and the decimation factor . Next, we consider a compress- and-estimate (CE) sour ce coding scheme, in which the encoder first compresses the decimated sequence subject to an MSE criterion (with respect to the decimated sequence), and the original random walk is estimated only at the decoder . W e ev aluate the distortion under CE in a closed form and show that there exists a non- zero gap between the distortion under the two schemes. This differ ence in performance illustrates the importance of having the decimation factor at the encoder . Index T erms — Indirect source coding, Gaussian ran- dom walk, Wiener process I . I N T RO D U C T I O N Consider the situation in which one is interested in compressing or transmitting a data sequence X N = ( X 1 , . . . , X N ) , generated by a source X , but can only access its factor M decimated version Y n = X M n , n = 1 , . . . , N / M . X N ↓ M Est Enc Y N M e X N Dec b X N k x − b x k 2 { 0 , 1 } b N R c Fig. 1: Estimate-and-compress (EC) strategy . The decimated data is first used to obtain an optimal MSE estimate of the source sequence X N . The encoder then describes this estimate to the decoder using N R bits. Assuming a compression or communication rate of R bits per symbol to describe X , the encoder has at most RN bits to represent Y N M , where N M , N / M . W ithout loss of generality , we assume here and throughout the paper that N M is an integer . The problem of finding the N R bit representation that minimizes the MSE with re- spect to X N is known as the indir ect (or remote) source coding problem. Classical results in source coding [1], [2], [3], [4] show that optimal compression is achieved by an estimate-and-compress (EC) strategy: the encoder first computes e X N , the optimal estimate of X N from its decimated version Y N M , and then compresses e X N under a MSE criterion subject to the bit constraint (Fig. 1). The resultant expected MSE is called the indirect DRF of X N gi ven Y N M , and we denote it here by D EC ( M , R ) . As N goes to infinity , D EC ( M , R ) is a function of only the bitrate R and the decimation factor M [4]. This work focuses on the fact that, in man y situations, the encoder cannot compute e X N . This may be the result of an unkno wn decimation factor M or a lack of computing resources. When the encoder is unable to estimate X N prior to encoding, a different scheme known as compr ess-and-estimate is often employed [5]. In this scheme, depicted in Fig. 2, the observed decimated sequence Y N M is encoded in an optimal manner subject to an MSE criterion. The original sequence X N is estimated at the decoder from the compressed version of Y N M . The distortion under X N ↓ M Enc Dec Y N M { 0 , 1 } b N R c Est b X N k x − b x k 2 b Y N M Fig. 2: Compress-and-estimate (CE) strate gy . The observed sequence Y N M is encoded so as to minimize the MSE between Y N M and b Y N M . The source sequence X N is estimated from the output of the decoder . this scheme is denoted as D CE ( M , R ) and provides an upper bound for the distortion under EC. That is, it bounds from abo ve the minimal distortion in the indirect source coding problem of X N gi ven Y N M when the decimation factor M is unkno wn at the encoder . In this paper we focus on the case where the source X is a standard Gaussian random walk, defined as X n = n ∑ i = 1 W i , n = 1 , 2 , . . . , (1) where W 1 , . . . , W n are standard normal and independent of each other . The process (1) arises as the uniform samples of the W iener process [6], or the discrete- time W iener process. Applications of the random walk are many , ranging from diffusion models in physics to option pricing in financial mathematics [7]. The main contributions of this paper are closed form expressions of the distortion functions D EC ( M , R ) and D CE ( M , R ) for the Gaussian random walk X defined by (1). These expressions fully characterize the fundamental limit arising from representing a Gaussian random walk by quantizing its samples at a fixed bitrate, as is necessary when, for example, transmitting them ov er a rate-limited link. Moreover , we show that for any decimation factor M > 1 and bitrate R > 0, the distortion under CE is strictly sub- optimal compared to the minimal distortion achieved by EC. That is, the scheme that encodes the decimated sequence Y so as to recov er Y with minimal distortion, attains distortion in recov ering X that is strictly lar ger than the scheme that encodes Y so as to recover X with minimal distortion. This result illustrates that, in problems in volving inference from lossy compressed information, the optimal lossy compression procedure depends on the end inference problem. As a result, ad-hoc lossy compression techniques that do not take into account the final inference procedure are necessarily sub-optimal. Ne vertheless, our results re veal that the difference between D EC ( M , R ) and D CE ( M , R ) is relati v ely small, and may be insignificant in many applications. This paper is organized as follo ws. In Sec. II we define our general source coding problem and the EC and CE schemes. In Sec. III we revie w rele vant kno wn results with respect to our general indirect source cod- ing problem. In Sec. IV we characterize the distortion under the EC and CE schemes. In Sec. V we ev aluate the resulting distortion expressions numerically and deri ve conclusions regarding the loss of performance when the decimation factor is unkno wn to the encoder . Finally , we pro vide concluding remarks and discuss future work in Sec. VI. I I . P R O B L E M F O R M U L A T I O N W e consider the source coding problem described in Figs. 1 and 2. In this problem, the standard Gaussian random walk X of (1) is decimated by a factor M to yield the process Y . Note that Y is also a Gaussian random walk, though with variance M rather than unit v ariance. For a time horizon N , the vector Y N M is encoded using the encoder f : R N M → { 0 , 1 } N R . (2) The decoder g : { 0 , 1 } N R → R N recei ves the binary w ord at the output of the encoder and pro vides a recon- struction sequence b X N = g ( f ( Y N M )) . The distortion is defined as the normalized MSE between X N and b X N : D f , g , 1 N N ∑ n = 1 E  X n − b X n  2 . Our goal in this paper is to characterize D f , g in the limit as N → ∞ under two different of encoder and decoder designs: the optimal design which has knowledge of the decimation factor at the encoder and decoder , and a suboptimal design which does not hav e this knowledge at the encoder . A. Optimal Sour ce Coding via EC The optimal source coding performance with respect to the source coding problem of Fig. 3 is defined as the minimum of D f , g under all pairs of encoders and de- coders. Since the encoder in this problem has no direct access to the signal X N it aims to accurately represent, the characterization of this minimal distortion is an indirect source coding problem [8, Ch. 3.5]. Classical results in source coding show that the minimum of D f , g is attained by the EC strategy illustrated in Fig. 1. That is, the encoder first estimates X N from the observ ed signal Y N M , and then compresses this estimated version in an optimal manner as in classical source coding [1], [3], [2]. For this reason, we set D EC ( M , R ) , lim inf N → ∞ inf f , g D f , g . X N ↓ M Enc Y N M Dec b X N { 0 , 1 } b N R c Fig. 3: Decimation and source coding setting. D EC ( M , R ) is called the indirect DRF of the process X gi ven the process Y as it describes the asymptotic optimal performance in indirect source coding. B. CE Sour ce Coding The CE scheme is defined by a particular sequence of encoders that generally differ from the optimal one used in EC. Specifically , the encoder f CE in CE is a minimum distance encoder with respect to a set of 2 N R code words drawn from the distribution that attains the DRF of the Gaussian vector Y N M at bitrate not exceeding R . The decoder g CE recei ves the index of the code word ˆ y N M nearest to the input sequence and outputs ˆ x N , obtained by linearly interpolating ˆ y N M as in b x n = M − n M b y n − + n M b y n + , n = 1 , . . . , N , (3) where n + = d n M e and n − = b n M c . Note that in the CE setting, although the encoding is optimal with respect to Y N M , it is not necessarily optimal with respect to X N . Ho wev er , the decimation factor M is not used by the encoder in CE, and hence this scheme may be useful when M is unkno wn. A distortion D is said to be achiev able under CE if there exists a sequence of encoders of the form f CE such that D f , g con v erges to D as N → ∞ . W e denote by D CE ( M , R ) the infimum ov er all achiev able distortions under CE. I I I . B AC K G R O U N D In this section we revie w relev ant known results for encoding the Gaussian random walk X N of (1). Since X N is Gaussian and Marko vian, the minimal MSE (MMSE) estimate of X N from Y N M is simply the interpolation of the decimated version. That is e X n , E  X n | Y N M  = M − n M Y n − + n M Y n + , n = 1 , 2 , . . . , where n + = d n M e and n − = b n M c . The resulting MMSE, which we denote by mmse ( M ) , is giv en by mmse ( M ) , 1 N N ∑ n = 1 E  X n − e X n  2 = M − M − 1 6 . Note that due to the properties of conditional expecta- tion, for any encoder f : R N M →  1 , . . . , 2 N R  we have mmse  X N | f ( Y N M )  = mmse ( M ) + mmse ( e X N | f ( Y N M )) . (4) Therefore, mmse ( M ) is a tri vial lower bound to the functions D EC ( M , R ) and D CE ( M , R ) . Moreover , as explained in [2], it follows from (4) that the minimal distortion in estimating X N from any N R -bit representation of Y M / N is attained by the optimal encoding of e X N subject to this bit constraint. Hence, the EC scheme, in which the encoder first estimates e X N and then encodes it, is optimal. Another , trivial lower bound to D EC ( M , R ) and D CE ( M , R ) is giv en by the (standard) DRF of the process X . This DRF is defined as the limit infimum as N → ∞ of the normalized distortion of the Gaus- sian vector X N . The latter is giv en via Kolmogoro v’ s expression [9] D X N ( θ ) = 1 N N ∑ k = 1 min [ θ , λ k ] (5a) R X N ( θ ) = 1 2 N N ∑ k = 1 max [ 0 , log ( λ k / θ )] , (5b) where λ k ’ s are the eigen v alues of the covariance matrix Σ X N of X N . In our case of X N as a standard Gaussian random walk, Berger [10] showed that λ k = h 2 sin  2 k − 1 2 N + 1 π 2 i − 2 , k = 0 , ..., N − 1 , (6) and concluded, upon taking the limit in (5), that D X ( R θ ) = Z 1 0 min { θ , S ( φ ) } d φ (7a) R θ = 1 2 Z 1 0 max { 0 , log ( S ( φ ) / θ ) } d φ , (7b) where S ( φ ) = ( 2 sin ( π φ / 2 )) − 2 is the asymptotic den- sity of the eigen v alues of Σ X N . I V . D I S T O RT I O N U N D E R E C A N D C E W e now deri ve our main results by characterizing the distortion under EC and CE in recovering the random walk X from its decimated version Y . A. Estimate-and-Compr ess From the definition of D EC ( M , R ) and the decompo- sition (4), it follo ws that D EC ( M , R ) = mmse ( M ) + lim inf N → ∞ inf f mmse ( X N | f ( Y N M )) = mmse ( M ) + D e X ( R ) , (8) where D e X ( R ) is the DRF of the process e X . There- fore, characterizing the distortion in EC is obtained by solving a source coding problem with respect to e X . No w the process B defined as B n ,  X n − e X n  returns to zero at least every M steps and has av erage v ariance equals to mmse ( M ) . Hence the variance of e X n = X n − B n increases at the same rate as the variance of X n , and Berger’ s coding theorem for X n [10] can be applied to e X n . Therefore, the DRF of e X is given by the limiting expression for the DRF of the Gaussian vector e X N , using K olmogorov’ s e xpression (5) leading to the follo wing result: Theorem 1: Let e S ( φ ) , ( 2 sin ( φ π / 2 )) − 2 − 1 − M − 2 6 . Then the indirect DRF of the random walk X given its factor M decimated version Y equals D EC ( M , R θ ) = mmse ( M ) + M Z 1 0 min { θ , e S ( φ ) } d φ (9a) R θ = 1 2 M Z 1 0 max n 0 , log  e S ( φ ) / θ o d φ , (9b) Pr oof: W e show in the Appendix that the N M non-zero eigen v alues of the cov ariance matrix of e X N are given by e λ k ( M ) = M 2 h 2 sin  ( 2 k − 1 ) M 2 N + 1 π 2 i − 2 − M 2 − 1 6 , k = 0 ... N M − 1 (10) Substituting (10) into (5) we hav e D e X N ( R θ ) = 1 M M N N M ∑ k = 1 min [ θ , e λ k ] (11a) R θ = 1 M M 2 N N M ∑ k = 1 max [ 0 , log ( e λ k / θ )] , (11b) where the e λ k ’ s are giv en by (10). T aking the limit in (11a) as N → ∞ with k M / N → φ ∈ ( 0 , 1 ) and θ 0 = θ / M leads to the integral representation for D e X ( R ) . Finally , (9) is obtained by adding the MMSE term to D e X ( R ) . B. Compr ess-and-Estimate W e now consider the compress-and-estimate scheme. As in EC, we begin from the decomposition in (4). Ho wev er , instead of using the optimal encoder that attains the DRF of e X , we use the encoder f CE that maps Y N M to one of 2 N R possible sequences b y N M ( 1 ) , . . . , b y N M ( 2 N R ) . By linearity of e X N in Y N M , we hav e that the MMSE estimate of X N from f EC ( Y N M ) is gi ven by the interpolation (3), hence mmse  X N | b Y N M  = mmse  X N | f CE  Y N M  , where b Y N M = g EC  f CE ( Y N M )  . Therefore, in order to deri ve D CE via (4), it is left to characterize the term mmse ( e X N | f CE ( Y N M )) . Note that unlike in EC, this term does not describe a distortion under optimal encod- ing, since while optimal encoding was performed, it was performed with respect to Y N M rather that e X N . Therefore, we characterize mmse ( e X N | f CE ( Y N M )) by expressing it in terms of the error in encoding Y N M with respect to the CE codebook: ε , Y N M − ˆ Y  f CE ( Y N M )  , This connection is achie ved by the following lemma: Lemma 2: For any N , M , and encoder f we ha ve: mmse ( e X N | b Y N M ) = 2 M 2 + 1 3 N M N M + 1 ∑ n = 1 E [ ε 2 n ] + M 2 − 1 3 N M N M ∑ n = 1 E [ ε n ε n + 1 ] − 2 M 2 + 3 M + 1 6 N M E [ ε 2 N M ] . (12) The proof of Lem. 2 can be found in the Appendix. Using Lem. 2 with the encoder f CE , we obtain a closed-form e xpression for D CE ( M , R ) , as per the follo wing theorem: Theorem 3: For any decimation factor M and bi- trate R , the infimum over all acheiv able distortions using the CE scheme is giv en by D CE ( M , R θ ) = mmse ( M ) + 2 M 2 + 1 3 M Z 1 0 min { S ( φ ) , θ } d φ + M 2 − 1 3 M Z 1 0 min { S ( φ ) , θ } cos ( π φ ) d φ , (13a) R θ = 1 2 M Z 1 0 max [ 0 , log ( S ( φ ) / θ )] d φ , (13b) where S ( φ ) = ( 2 sin ( π φ / 2 )) − 2 , as in (7). Pr oof: Only a sketch of the proof is provided here. The full proof can be found in the Appendix. In vie w of (4), it is enough to sho w that mmse  e X N | f CE ( Y N M )  con v erges to the water -filling part in (13). Using Lem. 2 with f CE implies that the first term in the RHS of (12) con v erges to 2 M 2 + 1 3 M D X ( M R ) , and leads to the first term in (13a). In order to ev aluate the term E [ ε n ε n + 1 ] in (12), we consider the properties of the encoder f CE . The joint distribution of the two sequences Y N M and ˆ Y N M , at the input and output of the encoder , respecti v ely , beha v es as if both sequences were drawn from the joint P ∗ Y N M , ˆ Y N M that attains the DRF of the vector Y N M [11], [12]. In our case, this distribution is defined by a Gaussian channel P ˆ Y N M , Y N M . Therefore, by setting ˆ ε , Y − b Y , we conclude that E [ ε n ε n + 1 ] = E [ ˆ ε n ˆ ε n + 1 ] = N M ∑ k = 1 u k [ n ] u k [ n + 1 ] , where u k ’ s are the eigen v ectors of cov arience matrix Σ Y N M , gi ven in [10]. The beha vior of the last term in the limit N → ∞ leads to the second term in (13a). V . A N A LY S I S A N D I N T E R P R E TA T I O N S Since the parameter θ obscures the direct depen- dency of D EC and D CE on R , we will consider the con- ditions under which we can eliminate the parameter θ . W e will then numerically analyze the dual dependency of D EC and D CE and M and R . A. High Rate Characterizations When the number of bits per decimated symbol M R is large, θ can be eliminated from (9) and (13), leading to single-line expressions for D EC ( M , R ) and D CE ( M , R ) . This leads to the following proposition. Proposition 4: (i) For RM ≥ 1, D CE ( M , R ) = mmse ( M ) + 2 M 2 + 1 3 M 2 − 2 M R (14) (ii) For M R ≥ log 2 h 1 + 3 / q 3 + 6 M 2 i ≥ 1, D EC ( M , R ) = mmse ( M ) (15) + 1 +  2 + q 3 + 6 M 2  M 2 6 M 2 − 2 M R The proof of Prop. 4 can be found in the Appendix. Note that, as expected, (15) and (14) reduce to D X ( R ) for M = 1. It follows from Prop. 4 that D CE ( M , R ) − D EC ( M , R ) = 1 +  2 − q 3 + 6 M 2  M 2 6 M 2 − 2 M R , (16) whene ver M R ≥ log 2 h 1 + 3 / q 3 + 6 M 2 i . B. Discussion Figs. 4 and 5 show the distortion expressions as functions of the bitrate R and the decimation factor M , respectiv ely . W e see that both D EC ( M , R ) and D CE ( M , R ) are bounded from below by D X ( R ) and mmse ( M ) , representing the minimal distortion only due to lossy compression and decimation, respectively . Further , both D EC ( M , R ) and D CE ( M , R ) approach the Fig. 4: Distortion as a function of the bitrate R for a fixed downsampling factor M = 100. The bottom figure sho ws the difference D CE ( M , R ) − D EC ( M , R ) . bounds in the two extremes of no decimation ( M = 1 ) and infinite bitrate ( R → ∞ ). That is, for lo w values of R compared to M , the distortion under both schemes is dominated by the error due to lossy compression, whereas the distortion is dominated by interpolation error when R is large compared to M . The bottom of Figs. 4 and 5 illustrate the perfor- mance gap in using CE compared to EC, gi ven by (16) for suf ficiently large M R . This gap is maximal in the transition region between rate-dominant and decimation-dominant distortion. As can be seen in Fig. 4, although the performance gap is positive for any M and R , it is relativ ely small. For example, when M = 100, the maximal value of (16), i.e. the performance loss from using CE instead of the optimal EC, is 2 . 7%, and thus CE may be used as a near approximation to optimal performance when EC is impractical or , due to an ignorance of M at the encoder , impossible. Fig. 5: Distortion as a function of downsampling factor M for a fixed R = 0 . 01 bits per symbol of X . The bottom figure shows D CE ( M , R ) − D EC ( M , R ) . V I . C O N C L U S I O N S W e have deri ved a closed-form expression for the minimal distortion in recovering a Gaussian random walk from a finite-bit representation of its decimated version. This expression quantifies the beha vior of the minimal distortion subject to decimation and lossy compression. This expression also confirms the fol- lo wing expected behavior: con ver gence to the standard DRF of the random walk at low coding rates where encoding error dominates; an interpolation error floor at high coding rates where decimation error dominates; and increased degradation with increasing decimation. In addition, we considered the distortion in re- cov ering the random walk under a CE scheme. In this scheme, the encoding of the decimated process is done with respect to a random codebook derived from its rate-distortion achie ving distrib ution. That is, a codebook that is designed to attain the DRF of the decimated process, rather than the original (not decimated) random walk. In particular , the encoder in this scheme is not informed of the decimation factor . The comparison between the two distortion expressions provides the excess distortion as a result of using the sub-optimal CE encoding. In particular , it provides a distortion bound on the price of an unkno wn decimation factor at the encoder . W e show that this price is small and need be considered only for a narro w band of R and M values where neither source of distortion, i.e. neither the bit constraint nor the decimation, hav e become dominant distortion factors. A C K N O W L E D G M E N T S This research was supported in part by the NSF Center for Science of Information (CSoI) under grant CCF-0939370. R E F E R E N C E S [1] R. Dobrushin and B. Tsybakov , “Information transmission with additional noise, ” vol. 8, no. 5, pp. 293–304, 1962. [2] J. W olf and J. Ziv , “Transmission of noisy information to a noisy receiv er with minimum distortion, ” IEEE T rans. Inf. Theory , v ol. 16, no. 4, pp. 406–411, 1970. [3] H. W itsenhausen, “Indirect rate distortion problems, ” IEEE T r ans. Inf. Theory , vol. 26, no. 5, pp. 518–521, 1980. [4] A. Kipnis, A. J. Goldsmith, Y . C. Eldar , and T . W eissman, “Distortion rate function of sub-Nyquist sampled Gaussian sources, ” IEEE Tr ans. Inf. Theory , vol. 62, no. 1, pp. 401– 429, Jan 2016. [5] A. Kipnis, S. Rini, and A. J. Goldsmith, “Compress and estimate in multiterminal source coding, ” 2017, unpublished. [Online]. A vailable: https://arxiv .org/abs/1602.02201 [6] A. Kipnis, A. J. Goldsmith, and Y . C. Eldar , “The distortion-rate function of sampled Wiener processes, ” CoRR , v ol. abs/1608.04679, 2016. [Online]. A vailable: http://arxiv .org/abs/1608.04679 [7] G. W eiss, Aspects and Applications of the Random W alk, ” Random Materials and Processes, Series Eds. H. Stanle y and E. Guyon . North Holland, 1994. [8] T . Berger , Rate-distortion theory: A mathematical basis for data compr ession . Englew ood Clif fs, NJ: Prentice-Hall, 1971. [9] A. Kolmogorov , “On the Shannon theory of information transmission in the case of continuous signals, ” vol. 2, no. 4, pp. 102–108, December 1956. [10] T . Berger , “Information rates of Wiener processes, ” IEEE T r ans. Inf. Theory , vol. 16, no. 2, pp. 134–139, 1970. [11] R. Gallager , Information theory and r eliable communication . W iley , 1968. [12] I. K ontoyiannis and R. Zamir , “Mismatched codebooks and the role of entropy coding in lossy data compression, ” IEEE T r ans. Inf. Theory , vol. 52, no. 5, pp. 1922–1938, 2006. A P P E N D I X A. Eigen values of Σ e X N Here we sk etch our deri v ation of the eigen v alues for the cov ariance matrix of the down-sampled and interpolated sequence e X , as giv en in (10) and used in the characterization of the D EC in (11a). W e follow the same proof approach as used by Berger for the non-decimated random Gaussian walk in [10]. Let e Σ X N be the cov ariance matrix of the interpolated process e X N with eigen v alues e λ and eigen vectors e f . Under the assumption of unit variance, this gi ves us e λ e f ( i ) = i − − 1 ∑ j = 0 j e f ( j ) + i + − 1 ∑ j = i − e f ( j )  M − j M i − + j M i  + i N − 1 ∑ j = i + e f ( j ) e λ [ e f ( i + 2 ) − 2 e f ( i + 1 ) + e f ( i )] = 0 From this we conclude that the eigenv ectors f must be piece-wise linear . Howe v er , since the interpolation and do wnsampling factors are the same, the boundary conditions remain the same as in the uninterpolated Wiener process case. In [10], they were shown as e f ( 0 ) = 0 e λ [ e f ( n ) − e f ( n − 1 )] = e f ( n ) Thus to satisfy both the piece-wise linearity as well as the boundary conditions, we consider piece-wise linear interpolations of the decimated eigen v ectors of the original W iener process. That is, if we let λ and f be the eigen v alues and eigen v ectors of Σ X , the cov ariance of the un-decimated sequence X , as deri ved in [10], e f k ( n ) = B  n + − n M f k ( n − ) + n − n − M f k ( n + )  f k ( n ) = A sin  2 k − 1 2 N + 1 π n  e f k ( n ) = C  n + − n M sin h 2 k − 1 2 N + 1  π i − i + n − n − M sin h 2 k − 1 2 N + 1  π i + i  (17) where n + = d n M e M and n − = b n M c M , and A , C are normalization constants. Since each element of e X is the result of a linear combination of elements of the decimated sequence Y , the dimensionality of e X ≤ the dimentionality of Y , and thus (17) only holds for k ≤ N M . For k > N M , e λ k = 0, and thus we are unconcerned with the corresponding eigen v ectors. No w to determine the eigenv alues, let p , jM , where j is an integer . e f ( p ) = f ( p ) = A sin  2 k − 1 2 N + 1 π p  e λ [ e f ( p + 2 M ) − 2 e f ( p + M ) + e f ( p )] = − C 1 3 sin ( τ π ( p + M ))[( M 2 − 1 ) cos ( τ π M ) + 2 M 2 + 1 ] τ , 2 k − 1 2 N + 1 e λ k = C A ( M 2 − 1 ) cos ( τ π M ) + 2 M 2 + 1 12 M 2 sin 2 ( τ π 2 M ) e λ k = D 2 cos 2 ( τ π 2 M ) + 1 12 sin 2 ( τ π 2 M ) + 1 6 M 2 ! e λ k = M 2 h 2 sin  ( 2 k − 1 ) M 2 N + 1 π 2 i − 2 − M 2 − 1 6 , D , C , A = normalization constants This proves Eq 10. B. Pr oof of Lem. 2 W e here provide the complete proof for (12). Note that throughout the main paper , all sequences are treated as 1-index ed (that is, the first element is X 1 ). For the simplicity of the calculation, the following deriv ation is done for a 0-indexed sequence (thus the first element is X 0 ). Ho we ver , by simply re-indexing the final result, we achie ve (12). mmse ( e X | b Y ) = 1 N N − 1 ∑ n = 0 E  e X n − E [ e X n | b Y ]  2 = 1 N N M − 1 ∑ n = 0 ( n + 1 ) M − 1 ∑ m = nM E  e X − E [ e X | b Y ]  2 = 1 N N M − 1 ∑ n = 0 ( n + 1 ) M − 1 ∑ m = nM E h e X − ( n + 1 ) M − m M b Y n − m − nM M b Y n + 1 i 2 b Y n = Y n − ε n = 1 N N M − 1 ∑ n = 0 ( n + 1 ) M − 1 ∑ m = nM E h ( n + 1 ) M − m M ε n + m − nM M ε n + 1 i 2 = 1 N M N M − 1 ∑ n = 0 2 M 2 + 3 M + 1 6 E [ ε 2 n ] + M 2 − 1 3 E [ ε n ε n + 1 ] + 2 M 2 − 3 M + 1 6 E [ ε 2 n + 1 ] = 1 N M N M ∑ n = 0 h 2 M 2 + 1 3 E [ ε 2 n ] i + N M − 1 ∑ n = 0 h M 2 − 1 3 E [ ε n ε n + 1 ] i − 2 M 2 − 3 M + 1 6 E [ ε 2 0 ] − 2 M 2 + 3 M + 1 6 E [ ε 2 N M ] ! E [ ε 2 0 ] = 0 = 2 M 2 + 1 3 N M N M ∑ n = 0 E [ ε 2 n ] + M 2 − 1 3 N M N M − 1 ∑ n = 0 E [ ε n ε n + 1 ] − 2 M 2 + 3 M + 1 6 N M E [ ε 2 N M ] Re-indexing from 1 rather than 0 yields (12). C. Con ver gence of E [ ˆ ε n ˆ ε n + 1 ] In order to e v aluate the cross term in (12), first note that 1 N M N M − 1 ∑ n = 1 u k [ n ] u k [ n + 1 ] = 1 N M N M − 1 ∑ n = 1 A 2 k sin  2 k − 1 2 N M + 1 π 2 n  sin  2 k − 1 2 N M + 1 π 2 ( n + 1 )  = 1 N M N M − 1 ∑ n = 1 A 2 k sin  2 k − 1 2 N M + 1 π n  sin  2 k − 1 2 N M + 1 π ( n + 1 )  = A 2 k 2 cos  2 k − 1 2 N M + 1 π  + o ( 1 ) , where o ( 1 ) N → ∞ − → 0, and this last transition is because N M − 1 ∑ n = 1 cos  2 k − 1 2 N + 1 π ( 2 n + 1 )  is bounded in N M . Similarly , we ha ve 1 = N M ∑ n = 1 ( u k [ n ]) 2 = N M ∑ n = 1 A 2 k sin 2  2 k − 1 2 N M + 1 π n  = A 2 k N M N M N M ∑ n = 1  1 2 − 1 2 cos  2 2 k − 1 2 N M + 1 π n  , and hence A 2 k N M N → ∞ − → 2. As a result 1 N M N M − 1 ∑ n = 1 E [ ˆ ε n ˆ ε n + 1 ] = 1 N M N M − 1 ∑ n = 1 N M ∑ k = 1 u k [ n ] u k [ n + 1 ] min { θ , λ k } = 1 N M N M ∑ k = 1 min { θ , λ k } N M − 1 ∑ n = 1 u k [ n ] u k [ n + 1 ] = o ( 1 ) + 1 N M N M ∑ k = 1 N M A 2 k 2 min { θ , λ k } cos  2 k − 1 2 N M + 1 π  (18) W e now take the limit in (18) as N → ∞ with k / N M = kM / N → φ ∈ [ 0 , 1 ] . In this limit, the spectrum of Σ Y con v erges to M S ( φ ) , where S ( φ ) ,  2 sin  φ π 2  − 2 . Therefore, the sum in (18) con v erges to Z 1 0 min { θ , M S ( φ ) } cos ( π φ ) d φ = M Z 1 0 min  θ 0 , S ( φ )  cos ( π φ ) d φ , where θ 0 = θ / M . D. Pr oof of Pr op. 4 When θ ≤ 1 / 4, (7) reduces to D X ( R θ ) = θ R θ = log h 1 √ θ i (19) and hence D X ( R ) = 2 − 2 R . Since D CE ( M , R ) depends on the same asymptotic eigen value density S ( φ ) , we conclude from (13) that D CE ( M , R ) = M − M − 1 6 +  2 M 2 + 1 3 M  2 − 2 M R . For the function D EC ( M , R ) , the minimum of e S ( φ ) is 1 2 − 1 − M − 2 6 = 1 + M − 2 12 , and for θ smaller than this v alue we have D EC = M − M − 1 6 + M θ M R = 1 2 Z 1 0 log   2 sin ( φ π / 2 )  − 2 − 1 − M − 2 6  d φ − log ( θ ) 2 , = − 1 + log " 1 + r 1 − 4 1 − M − 2 6 # − log ( θ ) 2 . Eliminating θ from the last expression, leads to (15) D EC ( M , R ) = M − M − 1 6 +    1 +  2 + q 3 + 6 M 2  M 2 6 M    2 − 2 M R , which holds whenev er M R ≥ − 1 + log " 1 + r 1 + 4 1 − M − 2 6 # − log  1 + M − 2 12  = log √ 3 M + √ 5 M 2 − 2 √ 2 + M 2 . E. Pr oof of Thm. 3 In view of (4), it is enough to show that mmse  e X N | f CE ( Y N M )  con v erges to the water-filling part in (13). Usign Lem. 2 we have 2 M 2 + 1 3 M 2 N + M N mmse  Y N M | f CE ( Y N M )  N → ∞ − → 2 M 2 + 1 3 M 2 D Y ( M R ) = 2 M 2 + 1 3 M D X ( M R ) , which, using (7) with θ 0 = θ / M , leads to (13b) and to the first term in (13a). In order ev aluate the term E [ ε n ε n + 1 ] in (12), we consider the properties of the encoder f CE . The joint distribution of two sequence Y N M and ˆ Y N M , at the input and output of the encoder, respectiv ely , beha ves as if both sequences were drawn from the joint P ∗ Y N M , ˆ Y N M that attains the DRF of the v ector Y N M [11], [12]. In our case, this distrib ution is defined by Y N M = b Y N M + U Z N M , where U is the matrix of eigen v alues in the eigenv alue decomposition Σ Y = U Λ Y U H of Σ Y , and Z N M ∼ N ( 0 , Λ θ ) where ( Λ θ ) n , n = min { ( Λ Y ) n , n , θ } . The ro ws of U and entries of Σ Y are gi ven by [10]: λ k = σ 2 Y  2 sin  2 k − 1 2 N M + 1 π 2  − 2 u k [ n ] = A k sin  2 k − 1 2 N M + 1 π n  , (20) where k = 1 , ..., N M , σ 2 Y = M , and A k is a normalization constant. Let ˆ ε , Y − b Y . From the above, we conclude E [ ε n ε n + 1 ] = E [ ˆ ε ˆ ε n + 1 ] = N M ∑ k = 1 u k [ n ] u k [ n + 1 ] . W e showed in the Appendix C that 1 N N M − 1 ∑ n = 1 E [ ˆ ε n ˆ ε n + 1 ] N → ∞ − → Z 1 0 min  θ 0 , S ( φ )  cos ( π φ ) d φ , (21) and E [ ε 2 N M ] / N → 0. This expression leads to the second term in (13a). 