Transceiver Design with Low-Precision Analog-to-Digital Conversion : An Information-Theoretic Perspective

1 T ranscei ver Design with Lo w-Precision Analog-to-Digital Con v ersion : An Information-Theoretic Perspecti v e Jaspreet Singh, Student Member , IEEE, Onkar Dabeer , Member , IEEE, and Upaman yu Madho w , F ellow , IEEE Abstract Modern commun ication recei ver architectures center aro und digital signal pr ocessing (DSP), with the bulk of th e r eceiv er pro cessing being p erform ed on digital signals obtained a fter analo g-to-d igital conv ersion (ADC). In t his paper, we explore S han non-the oretic performan ce limits when ADC p recision is drastically red uced, from typical values of 8 - 12 bits u sed in curren t communication transcei vers, to 1 - 3 bits. The goa l is to obtain insight on whether DSP-centric transceiv er architectures are feasible as commun ication ba ndwidths scale u p, recogn izing that high-p recision ADC at high sampling rates is either un av ailable, or too co stly or power-hungry . Specifically , we e valuate the comm unication limits imposed b y low-precision ADC for th e ideal real discrete-time Additiv e White Gaussian Noise (A WGN) channel, under an average power con straint on the inpu t. For an ADC with K quantiza tion bins (i.e., a precision of log 2 K b its), we show t hat the Shann on capacity is achie vable by a discrete input distrib ution with at mo st K + 1 mass p oints. For 2 -bin ( 1 -bit) symmetric ADC, this result is tighten ed to show that bin ary antipo dal sign aling is optimum for any signa l-to-noise ratio ( SNR ). For multi-bit ADC, the capacity is c omputed nu merically , and the results ob tained are used to m ake the following encou raging observations regard ing system design with low-precision ADC : ( a) ev en at mo derately high SNR o f up to 20 dB, 2 - 3 bit qua ntization r esults in only 10 - 2 0 % red uction o f spectral efficiency , whic h is acceptable for large co mmunica tion ban dwidths, (b ) standar d eq uiproba ble p ulse amp litude m odulation with ADC th resholds set to implement m aximum likelihoo d har d dec isions is asymptotica lly optimum at h igh SNR , and works well at low to mo derate SNR s a s well. Jaspreet Singh and Upamany u Madho w are with the Department of El ectrical and Computer E ngineering, Uni versity of California, Santa Barbara. Their research was supported by the National Science Foundation under grants ANI-0220118 and ECS-0636621, and by the Office of Nava l Research under grant N00014-06-1-0066. E-mail: { jsingh, madhow } @ece.ucsb .edu Onkar Dabeer is with the S chool of T echnology and Computer Science, T ata Institute of Fundamental Research, Mumbai, India. His research was supp orted by Grant SR/FTP/E T A-16/2006 from the Department of Science and T echnology , Gov ernment of India, and in part by the Homi Bhabha Fellowship. E-mail: onkar@tcs.tifr .res.in DRAFT 2 Index T erms Channel Capacity , Optim um Inp ut Distribution, A WGN Channel, Analog- to-Digital Con verter . I . I N T R O D U C T I O N Digital signal processing (DSP) forms the core of modern digital communication receiv er implementati ons, with the analog baseband signal being con verted to digital form using Analog- to-Digital Con verter s (ADCs) whi ch typically ha ve 8-12 bits of precision. Operations such as synchronization, equalization a nd demodu lation are then performed in the digital dom ain, greatly enhancing the flexibility av ailable to the designer . The continuin g exponential advances in digit al electronics, often summarized by Moore’ s “law” [1], imply that integrated circuit implementati ons of such DSP-centric architectures can be expected t o contin ue scaling up i n speed and down in cost. Howe ver , as t he bandwidth of a communication sys tem i ncreases, accurate con version o f the analog receiv ed signal into digital form requires hi gh-precision, high-speed ADC, which is costly and power -hungry [2]. One pos sible approach for desi gning such hi gh-speed syst ems is to drastically reduce the num ber of bits o f ADC precision (e.g., to 1 -3 bits ) as sampling rates scale up. Such a design choice has significant impl ications for al l aspects of receiv er design, i ncluding carrier and timing synchronization, equalization, demodulation and decoding. Ho wev er , b efore embarking on a comprehensiv e rethinking of the communi cation system design, it is impo rtant to understand the fundam ental limits on communication performance impos ed by low-precision ADC. In this paper , we take a first step in this direction, in vestigating the Shannon-theoretic performance lim its for the following i dealized model: linear modulation over a real baseband Additive White Gaussian Noise (A WGN) channel with symbol rate Nyqui st s amples q uantized by a low-precision ADC. Th is induces a discrete- time memoryless A WGN-Quantized Output channel, which is depicted in Figure 1. Under an averag e power constraint on the inpu t power , we obtain t he following resul ts 1) For a K -level (i.e., lo g 2 K bits) output quantizer , we prove that the in put distribution need not have any more t han K + 1 mass points to achieve t he channel capacity . (Num erical computation of optim al input dis tributions reve als that K m ass points are sufficient.) An intermediate result of int erest is that, when the A WGN channel output is quantized with finite-precision, an a verage power constraint leads to an imp licit peak power constraint, in the sense that an op timal inpu t dis tribution mu st now have bounded support. DRAFT 3 ADC Quantizer Q + X Y N Fig. 1. Y = Q ( X + N ) : The A WGN-Quantized Ouput channel i nduced by the output quantizer Q . 2) For 1 -bit s ymmetric quantization , the preceding result can be tigh tened to show that binary antipodal signaling i s optimal for any signal-to-noise ratio ( SNR ). 3) For multi-bit quantizers, ti ght upper bou nds on capacity are obtained using a dual form ula- tion of the capacity problem. Near-optimal input distributions that app roach these bou nds are computed using the cutt ing-plane alg orithm [31]. 4) While the preceding results optimize the input distribution for a fi xed quantizer , comparison with an unquantized system requi res an opt imization ov er the choi ce of the quant izer as well. W e numericall y obtain optim al 2 -bit and 3 -bit symmetric quant izers. 5) From our numerical results, we infer t hat low-precision ADC incurs a relatively small loss i n spectral effi ciency compared to unq uantized observations. For example, 2 -bit ADC achie ves 95% of the sp ectral efficienc y attain ed w ith un quantized ob serv ations at 0 d B SNR . Even at a m oderately high SNR of 20 dB, 3 -bit ADC achiev es 85% of the spectral ef ficiency attained with unquantized observations. Th is i ndicates that DSP-centric design based on low-precision ADC is indeed attractiv e as com munication bandwidths scale up, since the small loss in spectral efficienc y should be acceptable in thi s regime. Further - more, we also o bserve that a “sensible” choice of standard equiprobable pulse amplit ude modulated (P AM) input with ADC thresholds set to im plement m aximum likelihood (ML) hard decisions achie ves performance which is quite close to that obtained by numerical optimizatio n of the qu antizer and i nput dist ribution. Related W o rk For a Di screte Memo ryless Channel (DM C), Gal lager first showed that the number of input points with nonzero probabi lity m ass need not exceed the cardinalit y o f the output [3, p. 96, DRAFT 4 Corollary 3]. In our s etting, the inp ut alphabet is n ot a priori discrete, and there is a power constraint, so that the result i n [3] does not apply . Our key result on the achiev ability of the capacity by a discrete input is actually an extension of a result of W itsenhausen in [4], where Dubins’ theorem [5] was used to show that the capacity of a (di screte-time, memoryless and stationary) c hannel with K output le vels, under a peak power constraint is achiev able by a discrete input with at m ost K points. The key t o our proof i s to show that under out put quant ization, an av erage power constraint im plies an im plicit peak powe r constraint, after wh ich we can use Dubins’ theorem in a mann er simi lar to the dev elopment in [4]. Prior work on the eff ect of reduced ADC precisio n on channel capacity with fixed input distribution includes [6], [7], [8]. Howe ver , other than our own preliminary results reported in [9], [10 ], we are not aw are of a Shannon-theoretic in vestigation wi th low-precision ADC that includes optimization of the inpu t dis tribution. While we are interested in fundamental limit s here, a strong motivation for this work comes from emergent applications in high-bandwid th, multiGi gabit, unlicensed wi reless commu nication systems using Ultrawideband (UWB) communicati on in the 3 - 10 GHz b and [11], and millimeter wa ve communication in the 60 GHz band [12]. Indeed, t here has been pri or exploration of t he impact of low-precision ADC in the specific context of UWB syst ems. Low power t ranscei ver architectures for UWB systems h a ve b een proposed in [13], [14]. The performance of UWB recei vers using 1 -bit ADC has been analyzed in [15], including the use of dither and oversam- pling. The ef fect of ADC precision on UWB performance is considered in [16]. Decomp osition of the UWB signal into parallel frequency channels in order to relax ADC speed requirements is consid ered in [17], [18]. Demodulation and int erference suppression techniqu es for UWB communication using 1 -bit ADC have been proposed i n [19]. Giv en the encouraging results here, it becomes important to e xplore the impact of low-precision ADC on recei ver tasks such as synchronization and equalization, which we have ignored i n o ur idealized model (essentially assuming that these tasks ha ve somehow already been accomplished). Related work on estimation using low- precision samples which may be relev ant for this purpose includes the use of di ther for signal reconstruction [20], [21], [22], frequency estimation using 1 -bit ADC [23], [24], choice of quant ization thresh old for s ignal ampli tude estimation [25], and signal parameter est imation us ing 1 -bit di thered quanti zation [26], [27]. DRAFT 5 Or ganization of the P aper The rest of the paper is organized as foll ows. The A WGN-Quantized Output channel m odel is described i n t he next section. In Section III, we show the existence of an i mplicit peak power constraint, and use it to prove that the capacity is achiev able by a dis crete input distribution. Section IV presents capacity computations, includi ng duality-based upper bounds on capacity . Quantizer optimization is con sidered in Section V, followed by the conclusions i n Section VI. I I . C H A N N E L M O D E L W e consider l inear m odulation over a real A WGN channel, with symbol rate N yquist sam- ples quant ized by a K -bin (or K -level) quantizer Q . Th is indu ces the following discrete-time memoryless A WGN-Quantized Ou tput (A WGN-QO) channel Y = Q ( X + N ) . (1) Here X ∈ R is th e channel inpu t with cum ulativ e distribution function F ( x ) , Y ∈ { y 1 , · · · , y K } is the (dis crete) channel output, and N is N (0 , σ 2 ) (the Gaussian rando m variable wit h m ean 0 and variance σ 2 ). Q maps the real v alued in put X + N to one of the K bins, producing a discrete output Y . In t his work, we only consider quantizers for which each bin is an interval of the real line. The quanti zer Q with K bi ns is therefore characterized by the set of its ( K − 1) thresholds q q q := [ q 1 , q 2 , · · · , q K − 1 ] ∈ R K − 1 , such that −∞ := q 0 < q 1 < q 2 < · · · < q K − 1 < q K := ∞ . The output Y is assigned the value y i when th e q uantizer inpu t ( X + N ) falls in th e i th bin, which is g iv en b y the int erv al ( q i − 1 , q i ] . The resulting transi tion probability functi ons are W i ( x ) = P ( Y = y i | X = x ) = Q  q i − 1 − x σ  − Q  q i − x σ  , 1 ≤ i ≤ K , (2) where Q ( x ) denotes t he complementary Gaussian distribution functi on Q ( x ) = 1 √ 2 π Z ∞ x exp( − t 2 / 2) dt . The Probabi lity M ass Function (PMF) of the output Y , corresponding t o t he i nput dis tribution F is R ( y i ; F ) = Z ∞ −∞ W i ( x ) dF ( x ) , 1 ≤ i ≤ K , (3) DRAFT 6 and the in put-output m utual informat ion I ( X ; Y ) , expressed explicitly as a function of F is I ( F ) = Z ∞ −∞ K X i =1 W i ( x ) log W i ( x ) R ( y i ; F ) dF ( x ) . 1 (4) Under an av erage power constraint P on the channel inp ut (i.e., E [ X 2 ] ≤ P ), we wish to compute the capacity of the channel (1), which is g iv en by C = sup F ∈F I ( F ) , (5) where F is th e s et of all distributions on R that satisfy the a verage power constraint, i.e., F =  F : Z ∞ −∞ x 2 dF ( x ) ≤ P  . (6) I I I . D I S C R E T E I N P U T D I S T R I B U T I O N A C H I E V E S C A P A C I T Y W e first employ the Karush-Kuhn-T ucker (KKT) optimality condition to show th at, even though we ha ve not imposed an explicit peak power constraint on the i nput, it is aut omatically induced by the average power constraint. Specifically , an opt imal in put dist ribution must have a bounded s upport set. This is then used to show that the capacity is achiev able b y a discrete inp ut distribution with at mo st K + 1 mass p oints. Note that ou r result does not, howe ver , guarantee that t he capacity is achieved by a unique in put distri b utio n. A. An Implicit P eak P ower Cons traint Using con vex optim ization principles, th e following necessary and sufficient KKT o ptimality condition can be deriv ed for our problem, in a manner si milar to the de velopment in [29], [30]. An input di stribution F is optim al for (5) if and o nly if there exists a γ ≥ 0 s uch that K X i =1 W i ( x ) log W i ( x ) R ( y i ; F ) + γ ( P − x 2 ) ≤ I ( F ) (7) for all x , with equality if x is in the support of F 2 , where the transition p robability functions W i ( x ) , and the output PMF R ( y i ; F ) are as specified in (2) and (3), respectively . The first term on the left hand si de of th e KKT condition is the Kullback-Leibler diver gence (or the relative entropy), D ( W ( ·| x ) || R ( · ; F )) , between the transit ion and the output distributions. 1 The logarithm is base 2 throughout the paper , so the mutual information i s measured in bits. 2 The support of F (or the set of increase points of F ) is the set S X ( F ) = { x : F ( x + ǫ ) − F ( x − ǫ ) > 0 , ∀ ǫ > 0 } . DRAFT 7 For con venience, let us denote i t by d ( x ; F ) . W e first study th e behavior of this function in t he limit as x → ∞ . Lemma 1: For the A WGN-QO channel (1) with in put distribution F , the d iv ergence functi on d ( x ; F ) satisfies the following properties (a) lim x →∞ d ( x ; F ) = − log R ( y K ; F ) . (b) There exists a finite cons tant A 0 such t hat ∀ x > A 0 , d ( x ; F ) < − lo g R ( y K ; F ) . Pr oof: W e ha ve d ( x ; F ) = K X i =1 W i ( x ) log W i ( x ) R ( y i ; F ) = K X i =1 W i ( x ) log ( W i ( x )) − K X i =1 W i ( x ) log ( R ( y i ; F ) ) . For any finite no ise v ariance σ 2 , as x → ∞ , the conditi onal PMF W i ( x ) tends to the unit mass at i = K . This observation, combined wi th the fact that t he entropy of a finite alphabet random v ariable is a con tinuous funct ion of it s probabi lity law , giv es lim x →∞ d ( x ; F ) = 0 − lo g( R ( y K ; F ) ) = − log( R ( y K ; F ) ) . T o prove part (b), we pick A 0 to be such th at W i ( A 0 ) < R ( y i ; F ) for i = { 1 , 2 , ..., K − 1 } , and also that W K ( A 0 ) > R ( y K ; F ) . Such an A 0 alwa ys exists because for x > q K − 1 , the transition probabilities W i ( x ) → 0 a nd are strictly mo notone decreasing functi ons of x for i = { 1 , ..., K − 1 } , while W K ( x ) → 1 and is a strictly mon otone increasing functi on of x (the strict monotoni city i s easy to see by ev aluating th e deriv atives of the transition probabiliti es). W ith such a choice of A 0 , we get that for x > A 0 , d ( x ; F ) = K X i =1 W i ( x ) log W i ( x ) R ( y i ; F ) < W K ( x ) log W K ( x ) R ( y K ; F ) < − log ( R ( y K ; F ) ) . Using Lemma 1 , we now prove the main result o f t his subsection. Pr oposition 1: F or the av erage power constrained A WGN-QO channel (1), an optimal in put distribution must hav e bou nded sup port. DRAFT 8 Pr oof: Let us assume that the i nput di stribution F ∗ achie ves 3 the capacity in (5 ), i .e., I ( F ∗ ) = C . Let γ ∗ ≥ 0 denote a correspondin g optim al Lagrange parameter , so t hat th e KKT condition is sati sfied. In other words, with γ = γ ∗ , and, F = F ∗ , (7) mu st be satisfied with an equality at ev ery point i n the support o f F ∗ . W e exploit this n ecessary con dition next to show that the support of F ∗ is upper bounded. Specifically , we prove that there exists a finite constant A 2 ∗ such t hat it i s not possib le to attain equality in (7) for any x > A 2 ∗ . Using L emma 1 , we first let lim x →∞ d ( x ; F ∗ ) = − log( R ( y K ; F ∗ )) = L , and also assume that there exists a finite constant A 0 such that ∀ x > A 0 , d ( x ; F ∗ ) < L . W e consider two possible cases. • Case 1: γ ∗ > 0 . If C > L + γ ∗ P , then pick A 2 ∗ = A 0 . Else pi ck A 2 ∗ ≥ max { A 0 , p ( L + γ ∗ P − C ) /γ ∗ } . In either sit uation, for x > A 2 ∗ , we get d ( x ; F ∗ ) < L , and, γ ∗ x 2 > L + γ ∗ P − C . This giv es d ( x ; F ∗ ) + γ ∗ ( P − x 2 ) < L + γ ∗ P − ( L + γ ∗ P − C ) = C . • Case 2: γ ∗ = 0 . Putting γ ∗ = 0 in the KKT condit ion (7), we get d ( x ; F ∗ ) = K X i =1 W i ( x ) log W i ( x ) R ( y i ; F ∗ ) ≤ C , ∀ x. Thus, L = lim x →∞ d ( x ; F ∗ ) ≤ C . Picking A 2 ∗ = A 0 , we therefore have that for x > A 2 ∗ d ( x ; F ∗ ) + γ ∗ ( P − x 2 ) = d ( x ; F ∗ ) < L. = ⇒ d ( x ; F ∗ ) + γ ∗ ( P − x 2 ) < C . Combining the two cases, we have shown t hat the support of the di stribution F ∗ has a finite upper bound A 2 ∗ . Using s imilar ar gument s, it can easily be shown that the support of F ∗ has a finite lower b ound A 1 ∗ as well, whi ch im plies t hat F ∗ has a bounded s upport. 3 That there exists an input which achie ves the supremum in (5) is shown in Appendix I. DRAFT 9 B. Ac hievability of Capacity by a Discr ete Inpu t In [4], W itsenhausen considered a stationary discrete-tim e mem oryless channel, with a contin- uous input X taking v alues on the compact interv al [ A 1 , A 2 ] ⊂ R , and a discrete output Y of finite cardinality K . It was sh o wn that if the channel transition probabilit y fun ctions are contin uous (i.e., W i ( x ) is continuous in x , for each i = 1 , · · · , K ), then the capacity is achiev able by a discrete input distribution wit h at most K m ass p oints. As stated in Theorem 1 belo w (proved in Appendix II), this result can be extended to show that, if an additiona l ave rage power constraint is impo sed on the input, the capacity is then achiev able by a discrete input with at most K + 1 mass points. Theor em 1: Consider a stati onary discrete-time memoryless channel with a cont inuous inp ut X that takes values in the bou nded interval [ A 1 , A 2 ] , and a di screte output Y ∈ { y 1 , y 2 , · · · , y K } . Let the channel transitio n probabilit y function W i ( x ) = P ( Y = y i | X = x ) be continuou s in x for each i , where 1 ≤ i ≤ K . The capacity of this channel, under an av erage power constraint on t he input, is achiev able by a di screte input distri b utio n wi th at mo st K + 1 mass points. Pr oof: See Appendix II. Theorem 1 , coupled with t he impli cit peak power constrain t deriv ed in the pre vious subsecti on (Proposition 1 ), gives us the fol lowing resul t. Pr oposition 2: The capacity of the avera ge po wer constrained A WGN-QO channel (1) is achie vable by a d iscrete input distribution wi th at m ost K + 1 points of support. Pr oof: Using notation from th e last subsection, let F ∗ be an opti mal dis tribution for (5), with the support o f F ∗ being cont ained in the bou nded interval [ A 1 ∗ , A 2 ∗ ] . Define F 1 to be the set of all aver age power const rained distributions F whose supp ort S X ( F ) is cont ained in [ A 1 ∗ , A 2 ∗ ] , i.e., F 1 = { F ∈ F : S X ( F ) ⊆ [ A 1 ∗ , A 2 ∗ ] } , (8) where F is the set of all av erage power constrained dist ributions on R , as defined in (6). Not e that F ∗ ∈ F 1 ⊂ F . Consi der the maxi mization of the mutual information I ( X ; Y ) over the set F 1 C 1 = max F ∈F 1 I ( F ) . (9) Since the transition probability functions in (2) are conti nuous in x , Theorem 1 impli es that a discrete distri b ution with at most K + 1 mass points achieves the m aximum C 1 in (9). Deno te DRAFT 10 such a di stribution by F 1 . Howe ver , since F ∗ achie ves the maximum C in (5) and F ∗ ∈ F 1 , it must also achieve th e m aximum in (9). T his im plies that C 1 = C , and that F 1 is optimal for (5), thus com pleting t he p roof. C. S ymmetric Inputs for Symmetr ic Quant ization For our num erical capacity computations ahead, we assume t hat the quantizer Q empl oyed i n (1) is symmetric, i.e., its threshol d vector q q q is sym metric about the ori gin. Giv en the sym metric nature of the A WGN no ise and the power constrain t, it seems intui tiv ely plausible that rest riction to symmetric quantizers should not be suboptim al from the point o f vie w of optim izing ov er the quantizer choice in (1), although a proof of this conjecture has elu ded us . H o weve r , once we assume that the qu antizer in (1) is symmetric, we can restrict attention to o nly symmetric input distributions without loss o f o ptimality , as stated in the following l emma. Lemma 2: If the quantizer in (1) is sym metric, th en, without loss of o ptimality , we can consider o nly symm etric in put distributions (i.e., F ( x ) = 1 − F ( − x ) , ∀ x ∈ R ) for the capacity computation in (5) . Pr oof: Suppose we are gi ven an input dis tribution F ( x ) th at is not necessarily s ymmetric. Consider now the foll owing sym metric mixt ure dist ribution ˜ F ( x ) = F ( x ) + 1 − F ( − x ) 2 . This m ixture can be achiev ed by choosing distribution F ( x ) or 1 − F ( − x ) with probabil ity 1 / 2 each. If we u se ˜ F ( x ) i n place of F ( x ) , the conditional entropy H ( Y | X ) remains unchanged due the symmetric nature of the noise N and t he quantizer . Howe ver , the output entropy H ( Y ) changes as follows. Suppose that, when F ( x ) is used, the PMF of Y is a a a = [ a 1 , ..., a M ] . Th en under 1 − F ( − x ) it i s ˆ a a a = [ a M , ..., a 1 ] . Hence under ˜ F ( x ) , the output Y has the m ixture PMF ˜ a a a = 1 2 ( a a a + ˆ a a a ) . Since ent ropy is a concav e functi on of the PMF , H ( Y )   Y ∼ ˜ a a a ≥ H ( Y )   Y ∼ a a a 2 + H ( Y )   Y ∼ ˆ a a a 2 = H ( Y )   Y ∼ a a a . It follows that under the sym metric d istribution ˜ F ( x ) , I ( X ; Y ) = H ( Y ) − H ( Y | X ) is greater than t hat under F ( x ) , wh ich proves the desired result. DRAFT 11 I V . C A P A C I T Y C O M P U TA T I O N W e no w consider capacity computation for the A WGN-QO channel. W e first pro vide an explicit capacity formula for the extreme scenario of 1 -bit symmetric quantization, and then discuss numerical com putations for m ulti-bit q uantization. A. 1 -bit Symmetric Quantizat ion : Binary An tipodal Si gnaling i s Opti mal W ith 1 -bit symm etric quantization, the channel is Y = s ign ( X + N ) . (10) Proposition 2 (section III-B) guarantees that the capacity of thi s channel is achie vable by a discrete input distribution wit h at most 3 points. This result is further tightened by the following theorem t hat shows th e o ptimality of binary antipo dal signaling for all SNR s. Theor em 2: For the 1 -bit sym metric quantized channel m odel (10), the capacity is achieve d by bi nary antipodal signaling and is given by C = 1 − h  Q  √ SNR  , SNR = P σ 2 , where h ( p ) is th e b inary entropy function h ( p ) = − p log( p ) − (1 − p ) log(1 − p ) , 0 ≤ p ≤ 1 . Pr oof: Since Y is binary it is easy to see t hat H ( Y | X ) = E  h  Q  X σ  , where E denotes t he expectation operator . Therefore I ( X , Y ) = H ( Y ) − E  h  Q  X σ  , which we wish to maximize ov er all input distributions satisfying E [ X 2 ] ≤ P . Since the quantizer is symmetri c, we can restrict attenti on to sy mmetric input distributions wit hout loss of op timality (cf. Lemma 2 ). On doing so, we obtain t hat the PMF of th e ou tput Y is also symmet ric (since the quantizer and the noise distribution are already symmetri c). Therefore, H ( Y ) = 1 bit, and we obtain C = 1 − min X symmetric E [ X 2 ] ≤ P E  h  Q  X σ  . DRAFT 12 Since h ( Q ( z )) i s an even function, we get t hat H ( Y | X ) = E  h  Q  X σ  = E  h  Q  | X | σ  . In Appendix III, we show that the funct ion h ( Q ( √ y )) i s con ve x in y . Thus, Jensen’ s in equality [32] implies t hat H ( Y | X ) ≥ h  Q  √ SNR  with equality iff X 2 = P . Coupled with the sym metry conditi on on X , this implies that binary antipodal signaling achi e ves capacity and the capacity is C = 1 − h  Q  √ SNR  . B. Multi-Bit Quantiz ation W e now consider K -leve l quantization, where K > 2 . It appears unlikely that closed form expressions for opt imal inp ut and capacity can be obtained, due to t he complicated expression for mutu al information. W e therefore resort to the cutting-plane algorith m [31, Sec IV -A] to generate o ptimal inp uts n umerically . For channels wi th cont inuous in put alph abets, th e cutting - plane algorithm can, in general, be used to generate n early o ptimal discrete input distributions. It is therefore well matched to our probl em, for which we already know that the capacity is achie vable by a discrete input distribution. It is worth mentionin g that discretized Blahut-Arimoto type algorithms to comp ute the capacity of infinite input finite (infinite)-output channels have earlier been reported in [43], although they do not incorporate an a verage power constraint on the input. W e fix the n oise variance σ 2 = 1 , and vary th e power P to obtain capacity at diffe rent SNR s. T o apply the cut ting-plane algo rithm, we take a fine quantized discrete grid on the interval [ − 10 √ P , 10 √ P ] , and optimize the inpu t distribution over t his grid. Not e that Proposit ion 1 (Section III-A) tells us that an optim al input distribution for our prob lem m ust have a bound ed support, but it does not give explicit values that we can use directly i n our s imulations . Howe ver , on employing th e cutt ing-plane algorit hm over th e int erv al [ − 10 √ P , 10 √ P ] , we find that the resulting input distributions ha ve supp ort sets well within thi s interval. Moreove r , increasing the interval length further does not change these results. DRAFT 13 The input di stributions generated by the cutt ing-plane alg orithm are s hown in o ur numerical results. W e find th at these di stributions h a ve support set cardinality less than K + 1 as predicted by Propositio n 2 . The optimali ty of th ese distributions can further be verified by comparing the mutual information th ey achiev e with easily computable tight upper bo unds on t he capacity . The computation of these upper bounds is discussed next. 1) Dual ity-Based Upper Bound on Channel Capacity: In the dual form ulation of the channel capacity problem, we focus o n the di stribution of the channel outp ut, rather than t hat of the input. Specifically , assume a channel wi th input alphabet X , t ransition law W ( y | x ) , and an a verage power constraint P . Then, for every choice of the o utput distribution R ( y ) , we hav e the following upper bound on the channel capacity C C ≤ U ( R ) = min γ ≥ 0 sup x ∈X [ D ( W ( ·| x ) || R ( · )) + γ ( P − x 2 )] , (11) where γ is a Lagrange p arameter , and D ( W ( ·| x ) || R ( · )) is the diver gence between the t ransition and output distri b utio ns. While [33] provides t his bound for a Di screte M emoryless Channel (DMC), it s extension to continuous alphabet channels has been established in [34], [35]. A detailed p erspectiv e on the use of duality-based upper bounds can be found in [36]. For an arbit rary choi ce of R ( y ) , the bou nd (11) might be quite loose. Therefore, t o obtain a tight upper boun d, we m ay need to ev aluate (11) for a lar ge num ber of output dis tributions and pick the minimu m o f the resulting upp er bounds. Thi s could b e tedi ous in g eneral, especially if the output alphabet is continuou s. Howe ver , for t he channel model we consider , the ou tput alphabet is dis crete with small cardinalit y . For example, for 2 -bit quantization, the sp ace of all output distributions is characterized by a set of ju st 3 parameters i n the interval (0 , 1) . This makes the dual formulation attractive , s ince we can easily obtain a tight upper bound on capacity by e valuating the upper bound i n (11) for different choices of these parameters. Next, we di scuss computation of the upper bou nd (11) for our problem, for a fixed outpu t distribution R ( y ) . Computation of the Upper Bound : For con venience, we denote d ( x ) = D ( W ( ·| x ) | | R ( · )) , and g ( x, γ ) = d ( x ) + γ ( P − x 2 ) , so that we need to comput e min γ ≥ 0 sup x ∈X g ( x, γ ) . Consider first the maximization over x , for a fixed γ . Altho ugh the in put alphabet X is the real line R , from a prac- tical standpoint, we can restrict attention to a bounded interval [ M 1 , M 2 ] while performing this maximization This is justified as follows. From Lemma 1 , we k now that lim x →∞ d ( x ) = log 1 R ( y K ) . DRAFT 14 The saturating nature of d ( x ) , coupled with the non -increasing nature of γ ( P − x 2 ) , impl ies that for al l practical purposes, the search for the supremum of d ( x ) + γ ( P − x 2 ) over x can be restricted to x ≤ M 2 , where M 2 is large enough to ensure that the dif ference | d ( x ) − log 1 R ( y K ) | is negligibl e for x > M 2 . In our sim ulations, we t ake M 2 = q K − 1 + 5 σ , where q K − 1 is the lar gest quantizer t hreshold, and σ 2 is the noi se variance. This choice of M 2 ensures t hat for x > M 2 , the conditi onal PMF W i ( x ) is nearly the same as the unit mass at i = K , which consequentl y makes the difference between d ( x ) and lo g 1 R ( y K ) negligible for x > M 2 , as d esired. Simi larly , the search for t he supremum over x can also be restricted to x ≥ M 1 = q 1 − 5 σ , where q 1 is the smallest q uantizer threshol d. Not e that if the quantizer and t he o utput dis tribution R ( y ) are picked to be s ymmetric, then the function g ( x, γ ) i s also symm etric in x , so that we can further restrict attention t o [0 , M 2 ] . W e now need to compute min γ ≥ 0 max x ∈ [ M 1 ,M 2 ] { g ( x, γ ) } . T o d o this, we q uantize the interval [ M 1 , M 2 ] to generate a fine grid { x 1 , x 2 , · · · , x I } , and approximate the maxi mization over x ∈ [ M 1 , M 2 ] as a maximization ov er this quantized grid. This reduce s the computation of the upper bound to com- puting the functi on min γ ≥ 0 max 1 ≤ i ≤ I g ( x i , γ ) . Denoting r i ( γ ) := g ( x i , γ ) , this becomes min γ ≥ 0 max 1 ≤ i ≤ I r i ( γ ) . Hence, we are left with the task of minimizing (over γ ) the maximum value of a finite set of functions o f γ , which in t urn can be done directly using the standard Matlab t ool f minimax . Moreover , we note that t he function being m inimized over γ , i.e. m ( γ ) := max 1 ≤ i ≤ I r i ( γ ) , i s con ve x in γ . This fol lows from the observation that each of the functions r i ( γ ) = d ( x i ) + γ ( P − x i 2 ) is con ve x in γ (i n fact, affine in γ ), so th at their pointwise maxi mum is also con vex i n γ [37, pp. 81]. The con ve xity of m ( γ ) guarantees that f minimax provides us th e global minimum ov er γ . 2) Numerical Results : W e n ow com pare num erical result s obtain ed using the cutting-plane algorithm with capacity upper bounds obtained using the preceding dual formulation. W e fix the choice of quant izer to 2 -bit sym metric quantization, in which case the quantizer is characterized by a single parameter q , with the q uantizer thresholds being {− q , 0 , q } . The results depicted in this section are for the particular quantizer choice q = 2 . The input d istributions generated by the cutting-pl ane algorithm at various SNR s (setti ng σ 2 = 1 ) are shown i n Figure 2, and the m utual i nformation achieved by th em is given in T able I. As p redicted by Propositio n 2 (section III-B), th e support set of the input dist ribution (at each SNR ) h as cardinality ≤ 5 . DRAFT 15 −8 −6 −4 −2 0 2 4 6 8 0 0.1 0.2 0.3 0.4 0.5 x PMF 0 dB −8 −6 −4 −2 0 2 4 6 8 0 0.1 0.2 0.3 0.4 0.5 X 5 dB −8 −6 −4 −2 0 2 4 6 8 0 0.1 0.2 0.3 0.4 0.5 10 dB −8 −6 −4 −2 0 2 4 6 8 0 0.1 0.2 0.3 0.4 0.5 15 dB Fig. 2. P robability Mass Function of the optimal input generated by the cutting-plane algorithm at v arious SNR s, for the 2-bit symmetric quantizer with thresholds {− 2 , 0 , 2 } . SNR ( dB ) − 5 0 5 10 15 20 Upper Bound 0 . 1631 0 . 40 55 0 . 8669 1 . 3859 1 . 51 27 1 . 5146 M I 0 . 1547 0 . 40 46 0 . 8668 1 . 3792 1 . 48 38 1 . 4839 T ABLE I D U A L I T Y - BA S E D U P P E R B O U N D S O N C H A N N E L C A PA C I T Y C O M PA R E D W I T H T H E M U T U A L I N F O R M A T I O N ( M I ) AC H I E V E D B Y T H E D I S T R I B U T I O N S G E N E R A T E D U S I N G T H E C U T T I N G - P L A N E A L G O R I T H M . For upper bound computations, we ev aluate (11) for diff erent symmetric outpu t d istributions. For 2 -bit quantization, the set of symmetric output s is characterized by just one parameter α ∈ (0 , 0 . 5 ) , with the probability dist ribution on t he out put being { 0 . 5 − α , α , α , 0 . 5 − α } . W e var y α over a fine discrete grid on (0 , 0 . 5) , and compute the upper b ound for each value of α . The least upp er bo und achieved thus, at a n umber of different SNR s, is shown in T able I From the results, we see that t he input distributions generated by the cutting-plane algo rithm are nearly opt imal, since they nearly achieve the capacity upper bound at e very SNR . It is also insightful t o lo ok at the KKT condition for these input distributions. For instance, cons ider an SNR of 5 dB, for which the input distri b utio n generated by th e cutting-plane algorit hm has support s et {− 2 . 86 , − 0 . 52 , 0 . 52 , 2 . 86 } and achieves a mutual i nformation of 0 . 86 68 bits . Figure 3 plots the function g ( x, γ ) (i.e., t he left hand side of the KK T condition (7)) for this in put, DRAFT 16 0 0.5 1 1.5 2 2.5 3 3.5 0.76 0.78 0.8 0.82 0.84 0.86 0.88 x g(x, γ ) [to test KKT condition] Fig. 3. KKT condition confirms the optimality of the input distribution generated by the cutting-plane algorithm. with γ = 0 . 1530 . W e see that g ( x, γ ) equals the mutual i nformation at point s i n the support set of t he input distribution, and is less t han the m utual in formation ev erywhere el se. The suffic ient nature of the KKT condition therefore confirms the opti mality of this i nput di stribution. Not e that we show th e p lot for x ≥ 0 only because g ( x, γ ) i s sym metric i n x . V . O P T I M I Z A T I O N O V E R Q U A N T I Z E R T ill now , we ha ve addressed the problem of capacity computation with a fixed output quant izer . The cutti ng-plane algorithm can be used to do this compu tation. In th is section, we consider quantizer optimizatio n, and nu merically ob tain optim al 2 -bit and 3 -bit symm etric quantizers. A Simple Benchmark: Wh ile an optim al quanti zer , along wi th a corresponding optimal input distribution, provides the absol ute commu nication li mits for our model, we do no t h a ve a sim ple analytical characterization of their depend ence o n SNR . From a sys tem designer’ s perspecti ve, therefore, it is o f interest to also examine suboptim al choices that are easy to adapt as a function of SNR , as long as the penalty relative to the optimal solution is not excessiv e. Specifically , we take the following in put and quantizer pair to be our benchmark strategy : for a K -lev el quantizer , consider equiprobabl e, equis paced K -P AM (Pulse Ampl itude Modu lation), with quantizer t hresholds chosen t o be th e mid-point s o f the inp ut mass po int locations. That is, the quantizer l e vels correspond to the ML hard decision boundaries. Both t he input mass poin ts DRAFT 17 and t he quantizer thresholds hav e a simple, well-defined dependence on SNR , and can therefore be adapted easily at t he recei ver based on the m easured SNR . A n explicit expression for the mutual information of our b enchmark s cheme is easy to comp ute. W e can also obtain in sight from the following lower bound on th e mu tual inform ation, which is a direct consequence of Fano’ s in equality [32, pp. 37]. H B ( X | Y ) ≤ h ( P e ) + P e log 2 ( K − 1) . = ⇒ I B ( X ; Y ) ≥ log 2 ( K ) − h ( P e ) − P e log 2 ( K − 1) , where h ( · ) is the binary entro py functi on, and the subscript B denot es the benchm ark choice. The probability o f error P e with the ML decisi ons is P e = 2  K − 1 K  Q r 3 SNR K 2 − 1 ! , where Q ( · ) is t he complementary Gaussian di stribution functi on. It is evident that as SNR → ∞ , P e → 0 , so that I B ( X ; Y ) → log 2 ( K ) bits. Th is im plies that the uniform P AM input with mid-point quanti zer t hresholds is near -optimal at high SNR . The issue to i n vestigate therefore is how much gain an opti mal quantizer and input p air provides over this benchmark at lo w to moderate SNR . Note that, for 1 -bit symmetric quantization, the benchmark input corresponds to binary antip odal s ignalling, which h as already been shown to be optimal for all SNR s. As before, we set th e nois e variance σ 2 = 1 for con venience. Of course, t he result s are scale- in va riant, in the sense that if both P and σ 2 are scaled by the same factor R (th us keeping the SNR unchanged), then there is an equiv alent quanti zer (obtained by scaling the thresholds by √ R ) that giv es i dentical p erformance. N U M E R I C A L R E S U LT S A. 2 -Bit Symmetric Quantizat ion A 2 -bit sy mmetric quantizer is characterized by a single p arameter q , with the quantizer thresholds being {− q , 0 , q } . W e therefore employ a brute force search over q to find an opt imal 2 - bit symm etric quantizer . In Figure 4, we plot the variation of the channel capacity as a function of the parameter q at various SNR s. Based on our simulations , we make the following observations DRAFT 18 0 1 2 3 4 5 6 7 0 0.5 1 1.5 2 Quantizer threshold ’q’ Capacity (bits / channel use) −5 dB 0 dB 3 dB 7 dB 10 dB 15 dB Fig. 4. 2-bit symmetric quantization : channel capacity (i n bits per channel use) as a function of the quantizer threshold q (noise variance assumed constant). • For any SNR , there is an optimal cho ice of q which maximizes capacity . For the benchmark quantizer (which is optimal at high SNR ), q scales as √ SNR , hence it is not surpri sing t o note that the o ptimal value o f q we obtain increases m onotonically wit h SNR at hig h SNR . • The plots show that the capacity varies quite slowly as a functi on of q . This is b ecause of the sm all variations in th e channel transiti on probabiliti es (2) as a fun ction o f q . • For any SNR , it is observed that, as q → 0 or q → ∞ , we approach the same capacity as with 1 -bit symm etric quantization (not shown for q → ∞ in the plo ts for 10 and 15 dB i n Figure 4). This conforms t o intuition : q = 0 reduces t he 2 -bit quantizer to a 1 -bit quantizer , while q → ∞ renders the thresholds at − q and q i nef fective i n distingu ishing between two finite v alued inputs, so that only the comparison with the quantizer threshold at 0 yields useful i nformation. Comparison with th e Benchmark : T able II compares the performance o f the preceding optim al solutions with the benchmark s cheme. Th e capacity with 1 -bit symmetri c quantization is also shown for ref erence. In addition to being nearly optimal at moderate to high SNR s, the benchmark scheme performs fairly well at low SNR s as well. For inst ance, ev en at - 10 dB SNR , which might correspond to a UWB system desig ned for very low bandwidth effic iency , i t achieve s 86% of the capacity achieved wi th optimal choice of 2 -bit quant izer and input dist ribution. On the other DRAFT 19 SNR (dB) − 20 − 10 − 5 0 3 7 10 15 1-bit optimal 0 . 0046 0 . 04 49 0 . 1353 0 . 3689 0 . 60 26 0 . 9020 0 . 9908 0 . 99 74 2-bit optimal 0 . 0063 0 . 06 13 0 . 1792 0 . 4552 0 . 69 32 1 . 0981 1 . 4731 1 . 93 04 2-bit benchmark 0 . 00 49 0 . 0527 0 . 1658 0 . 44 01 0 . 6868 1 . 0639 1 . 40 86 1 . 9211 T ABLE II 2 - B I T S Y M M E T R I C Q U A N T I Z A T I O N : M U T U A L I N F O R M A T I O N ( I N B I T S P E R C H A N N E L US E ) A C H I E V E D B Y T H E B E N C H M A R K S C H E M E , C O M PA R E D AG A I N S T T H E O P T I M A L S O L U T I O N S . 0 0.2 0.4 PMF 0 0.2 0.4 −q q 0 0.2 0.4 0 0.2 0.4 0 0.2 0.4 0 0.2 0.4 0 dB 4 dB 7 dB 10 dB 15 dB 20 dB −x 1 x 1 −x 1 x 0 x 1 −x 1 −x 2 x 2 x 1 X Fig. 5. 2 -bit symmetric quantization : optimal input distribution and qua ntizer at v arious SNR s (the dashed vertical lines depict the locations of the quantizer thresholds). hand, for SNR of 0 dB or above, the capacity is better than 95% of the optimal. These result s are encouraging from a practical standpoint, g iv en the ease o f implementing the benchmark scheme. Optimal Input Di stributions : It is interesting to examine the optimal input di stributions (given by the cutt ing-plane algorithm) correspondi ng to the o ptimal quantizers obt ained above. Figure 5 shows these distri butions, along with optimal quantizer threshol ds, for diff erent SNR s. The solid vertical lines show the locations of the i nput dist ribution points and their probabilities, whil e the quantizer t hresholds are depicted by t he dashed vertical lines. As expected, binary signalin g DRAFT 20 is found to be optimal for l ow SNR , since it would be difficult for t he receiv er to distingu ish between multiple input points located close to each other . The locations of the constellation points for the binary input are denoted by {− x 1 , x 1 } in the 0 dB plot in Figure 5. The number of m ass points increases as SNR is increased, with a new p oint (denot ed x 0 ) em er ging at 0 . On increasing SNR further , we see that th e poi nts {− x 1 , x 1 } ( and als o the quanti zer thresholds {− q , q } ) m ove farther apart, resul ting in increased capacity . Finally , when the SNR becomes enough that four input points can be disamb iguated, the point at 0 disappears, and we get two new points shown at { − x 2 , x 2 } . The ev entual con ver gence of thi s 4 -point constellati on to uniform P AM with mid- point quantizer th resholds (i.e., the benchmark scheme) is to be expected, since the benchmark scheme approaches the capacity b ound of two bits at high SNR. It is worth noting that the optimal in puts we obtained all have at mos t four points, even t hough Propositi on 2 (section III-B) is looser , guaranteeing the achiev ability of capacity by at mo st five points. B. 3-bit Symmetric Quantizat ion For 3 -bit symmetric quantization, we n eed to op timize over a space of 3 parameters : { 0 < q 1 < q 2 < q 3 } , with t he quantizer threshold s being {± q 1 , ± q 2 , ± q 3 } . Since brute force search is computationall y complex, we in vestigate an alternate iterative optimi zation procedure for jo int optimizatio n of t he i nput and the quantizer in this case. Specifically , w e begin with an i nitial quantizer choice Q 1 , and then it erate as foll o ws (st arting at i = 1 ) • For the quanti zer Q i , find an o ptimal inp ut. Call thi s input F i . • For the inp ut F i , find a locally optimal quanti zer , initiali zing the search at Q i . Call the resulting qu antizer Q i +1 . • Repeat the first two steps with i = i + 1 . W e terminate the process wh en the capacity gain between consecutive iterations becomes less than a small threshol d ǫ . Although the in put-output mutual in formation is a concave function al of the input distribution (for a fixe d quantizer), it i s not guaranteed to be concav e joi ntly over the i nput and the quantizer . Hence, the iterativ e procedure is not guaranteed t o p rovide an optim al input-qu antizer pair in general. A good choice of th e initial quantizer Q 1 is crucial to enhance the likelihood that it does conv erge to an optimal solution. W e discuss this next. DRAFT 21 SNR (dB) − 20 − 10 − 5 0 5 10 15 20 3-bit optimal 0 . 0069 0 . 06 67 0 . 1926 0 . 4817 0 . 97 53 1 . 5844 2 . 2538 2 . 83 67 3-bit benchmark 0 . 00 50 0 . 0557 0 . 1768 0 . 47 07 0 . 9547 1 . 5332 2 . 13 84 2 . 8084 T ABLE III 3 - B I T S Y M M E T R I C Q U A N T I Z A T I O N : M U T U A L I N F O R M A T I O N ( I N B I T S P E R C H A N N E L US E ) A C H I E V E D B Y T H E B E N C H M A R K S C H E M E , C O M PA R E D AG A I N S T T H E O P T I M A L S O L U T I O N S . High SNR Re gime : For high SNR s, we k now that the uniform P AM with mid-point quantizer thresholds (i.e., the benchmark schem e) is nearly op timal. Hence, this quanti zer is a good choice for initialization at high SNR s. The results we obt ain ind eed dem onstrate that this initializatio n works well at high SNR s. This i s seen by comparing the results of the iterative procedure with the results of a b rute force search over the quantizer choi ce (similar to the 2 -bit case considered earlier), as bot h of them provide alm ost identical capacity values. Lower SNRs : F or lower SN R s, one possibility is to try out different initializations Q 1 . Ho wev er , on trying out the benchmark initiali zation at some lower SNR s as well, we find t hat the iterative procedure still provides us wi th near optimal solutions (again verified b y comparing with brute force optimization results). While our results show that the iterativ e procedure (with b enchmark initializatio n) has provided (near) optim al solution s at d if ferent SNR s, we leav e the question of whether it wi ll conv erge in general t o an opti mal so lution or not as an open problem. Comparison with the Benchmark : The efficac y of th e benchmark initialization at lower SNR s suggests that the performance of the benchmark scheme sh ould not be too far from optim al at small SNR s as well. Th is is i ndeed the case, as s hown in T able III. At 0 dB SNR , for instance, th e benchmark schem e achi e ves 98% of the capacity achiev able with an op timal quanti zer choice. Optimal Input Distributions : The optimal input distributions and quantizers (obtained using the iterative procedure) are depicted i n Figu re 6. Binary ant ipodal signali ng is op timal at low SNR s (not shown). Increase in t he SNR first results in a new mass point at 0 , and subsequently in a 4 -point constellation. The trend is repeated, with the number of mass points increasing with SNR , till we get an 8 -point cons tellation which eventually moves towards uniform P AM, and the capacity approaches three bits. DRAFT 2 2 0 0.2 0.4 X PMF 0 0.2 0.4 0 0.2 0.4 0 0.2 0.4 0 0.2 0.4 0 0.2 0.4 0 0 3 dB 7 dB 10 dB 13 dB 15 dB 20 dB Fig. 6. 3 -bit symmetric quantization : optimal input distribution and qua ntizer at v arious SNR s (the dashed vertical lines depict the locations of the quantizer thresholds). Again, the optim al input distri b utio ns obt ained hav e at mo st K point s ( K = 8 ), while Proposition 2 in section III-B p rovides the l ooser guarantee that the capacity is achiev able with at most K +1 poin ts. Of course, the results abo ve are for the particular cases when the quantizers are also optimal (among symmetric quantizers), wherea s Proposition 2 holds for an y quantizer choice. Thus, it is pos sible that th ere might exist a K -lev el qu antizer for which the capacity is indeed achie ved by exactly K + 1 points. W e lea ve open, therefore, the question of whether the result in Proposition 2 can be t ightened to guarantee the achiev ability of capacity with at most K poi nts. C. Compa rison with Unqu antized Observat ions W e no w c ompare the capacity re sult s for dif ferent quantizer precisions against the capacity with unquantized observations (depicted in Fig ure 7). A sampling of these results i s provided i n T able IV. W e observe t hat at low SNR , the performance degradation due to low-precision quantization is small. For instance, at - 5 dB SNR , 1 -bit receive r quantizatio n achieves 68% of the capacity achie vable with infinite-precision, while wit h 2 -bit quantization , we can get as much as 90 % of DRAFT 23 −5 0 5 10 15 20 0 0.5 1 1.5 2 2.5 3 3.5 4 SNR (dB) Capacity (Bits/Channel Use) Infinite precision ADC 1−bit ADC 2−bit ADC 3−bit ADC Fig. 7. Capacity with 1-bit, 2-bit, 3-bit, and infinite-precision ADC . SNR (dB) − 10 − 5 0 5 10 15 20 1-bit ADC 0 . 0449 0 . 13 53 0 . 3689 0 . 7684 0 . 99 08 0 . 9999 0 . 9999 2-bit ADC 0 . 0613 0 . 17 92 0 . 4552 0 . 8889 1 . 47 31 1 . 9304 1 . 9997 3-bit ADC 0 . 0667 0 . 19 26 0 . 4817 0 . 9753 1 . 58 44 2 . 2538 2 . 8367 Unquantized 0 . 0688 0 . 1982 0 . 500 0 1 . 0286 1 . 7297 2 . 513 8 3 . 3291 T ABLE IV I M PAC T O F L OW - P R E C I S I O N A D C : C A PAC I T Y ( I N B I T S PE R C H A N N E L U S E ) W I T H D I FF E R E N T A D C P R E C I S I O N S , C O M PA R E D W I T H T H E U N Q UA N T I Z E D ( I N FI N I T E - P R E C I S I O N ) C A S E . the in finite-precision limit. This is t o be expected: if channel noise dom inates the actual signal, increasing the quantizer precision beyond a point does not help much in distingui shing between diffe rent sign al levels. The more surprising finding i s th at, e ven at moderately high SNR s, the loss due to low-precision sam pling remains quite acceptable. For example, 2 -bit quantization achie ves 85% of the capacity attained using unquantized observ ations at 10 dB SNR , while 3 -bit quantization achiev es 85 % of the unquantized capacity at 20 dB SNR . Encouraging resul ts of a similar nature hav e been report ed earlier in [6]. Howe ver , t he input alphabet there was kept fixed as binary t o begin wi th, so that th e good performance with low-precision receiver quantization is p erhaps less surpri sing. DRAFT 24 Spectral Efficiency (bits per channel use) 0 . 25 0 . 5 1 . 0 1 . 73 2 . 5 1-bit AD C − 2 . 04 1 . 79 − − − 2-bit AD C − 3 . 32 0 . 59 6 . 13 12 . 30 − 3-bit AD C − 3 . 67 0 . 23 5 . 19 11 . 04 16 . 90 Unquantized − 3 . 83 0 . 00 4 . 77 10 . 00 14 . 91 T ABLE V SNR ( I N D B ) R E Q U I R E D TO AC H I E V E A S P E C I FI E D S P E C T R A L E FFI C I E N C Y WI T H D I FF E R E N T A D C P R E C I S I O N S . While the loss in spectral effi ciency at fixed SNR is moderate, the loss in power efficienc y at fixed spectral ef ficiency is si gnificant (T able V). For example, if the sp ectral efficiency is fixed to that attained by an unquanti zed s ystem at 1 0 dB (which is 1 . 73 bit s/channel u se), then 2 -bit quantization incurs a loss of 2 . 30 dB. In practical term s, this p enalty in power i s m ore significant compared to the 15% loss in spectral efficienc y on using 2 -bit quantization at 10 dB SNR . This suggests, for example, that in order t o weather the impact of low- precision ADC, a moderate reduction in the spectral ef ficienc y is a better design choice than a n increase in the tra nsmi t power . V I . C O N C L U S I O N S Our Shannon -theoretic i n vestigation i ndicates that the us e of low-precision ADC is a feasible option for designing futu re hig h-bandwidth comm unication system s. The choice of low-precision ADC is con sistent w ith the overall s ystem desi gn goals for systems such as UWB and mm wa ve communication , where power is at a premium, d ue t o regulatory restrictions as well as due to the diffic ulty of generating lar ge transmit po wers with integrated circuits in low-cost silicon processes (e.g., see [38] for discussion of mm wa ve C MOS design ). Po wer -effi cient comm unication dictates the use of small const ellations, so that the sym bol rate, and hence the sampling rate, for a g iv en bit rate mus t b e high. This forces us tow ards using ADCs with lower precision, but fortunately , this is consistent wi th the use of small constellations in the first place for power -ef ficient design. Thus, if we plan on operating at low to moderate SNR , the small reduction in spectral ef ficiency due to l ow- precision ADC is acceptable in such systems , give n that b andwidth is pl entiful. There are several unresolved t echnical issu es that we lea ve as open problems. While we show that at most K + 1 poin ts are needed to achieve capacity for a K -le vel quantizer , our DRAFT 25 numerical resul ts show that at most K points are needed. Can this be proven, at least for symmetric quantizers? Are symmetric quantizers optimal? Does our iterative procedure (with the benchmark ini tialization, or some oth er judicious initialization ) for jo int optimizatio n of the input and th e quant izer con ver ge to an optimal solutio n in general? Are there ot her , provably optimal techniqu es with substantially lower complexity th an brute force search to perform this joint optimizatio n? A technical assum ption worth re visitin g is that of Nyquist samplin g (which induces the discrete-time m emoryless A WGN-Quant ized Output channel model considered in this work). While sym bol rate Nyq uist sampling is optimal for unquantized systems i n which t he transmit and receive filters are square root Nyquis t and the channel is i deal, for quant ized samples, we hav e obtained numerical results that show that fractionally spaced samples can actually lead to small performance gains. A detailed stu dy q uantifying s uch gains i s important in underst anding the tradeoffs between ADC speed and precision. Howe ver , we do not expect ove rsampli ng to play a significant role at low to m oderate SNR , giv en the sm all degradation in o ur Nyquist sampled system relative to unquant ized observa tions (for which Nyquist samp ling is indeed opti mal) in these regimes. Of course, oversampling in conjunction with hybrid analog/digital processing (e.g., using ideas analogous to delta-sigma quantization) could produce bigger performance gains , but this falls outsid e the scope of the present mod el. While our focus in this paper was on non-spread systems, it is known that l ow-pre cision ADC is often employed i n spread spectrum syst ems for l o w cos t implem entations [39]. In our prior examination of Shannon li mits for direct sequence spread spectrum sy stems with 1 -bi t ADC [9], we demons trated that binary signaling wa s suboptim al, but did not provide a complete characterization of an opti mal input distribution. Th e approach in the present paper impl ies t hat, for a spreading g ain G , a di screte in put di stribution with at m ost G + 2 points can achiev e capacity (although in practice, much smaller constellations would probably work well). Finally , we would like to emphasi ze that the Shannon-theoretic perspective provided in this paper is but a first step towards the design of communicatio n systems with low-precision ADC. Major technical challenges include the design of ADC-constrained methods for car rier and timing synchronization, channel equ alization, demodulation and decoding. DRAFT 26 A P P E N D I X I : A C H I E V A B I L I T Y O F C A PAC I T Y Theor em 3: [40], [41] Let V be a real normed linear vector space, and V ∗ be its norm ed dual space. (a) A weak* conti nuous real-valued functional f ev aluated on a weak* compact subset F of V ∗ achie ves it s maxi mum on F . (b) If i n addition t o part (a), F is a con vex subset, and f is a con ve x functional, then the maximum is achieved at an extreme point 4 of F . Pr oof: For part (a), see [40, p . 1 28, Thm 2]. Part (b) follows from the Bauer Maximum Principle (see, for example [41, p. 211]), whi ch holds since the dual space V ∗ , equipped with the weak* topology , is a locally con vex Hausdorff space [41, p. 205]. The use of part (a) o f the theorem to establish the existence of capacity-achie ving input distributions is st andard (s ee [30], [42] for details ). T o use t his t heorem for our channel mod el (1), we need t o show that the set F of all a verage p o wer constrained di stribution functions i s weak* compact, and the m utual inform ation functio nal I is weak* continuous over F , so that I achieves its maxi mum on F . T he weak* compactness o f F follows by [42, Lemma 3.1]. T o prove con tinuity , we need to show that F n w eak ∗ − − − → F = ⇒ I ( F n ) − → I ( F ) The finite cardinalit y of the outpu t for our problem trivially ensures this. Specifically , I ( F ) = H Y ( F ) − H Y | X ( F ) = − K X i =1 R ( y i ; F ) log R ( y i ; F ) + Z dF ( x ) K X i =1 W i ( x ) log W i ( x ) where, R ( y i ; F ) = Z ∞ −∞ W i ( x ) dF ( x ) . The conti nuous and bounded nature of W i ( x ) ensures t hat R ( y i ; F ) is continuous (by t he definition of weak* topology). Moreove r , the function K X i =1 W i ( x ) log W i ( x ) is also continuous a nd bounded, i mplying that H Y | X ( F ) i s al so continu ous (again by the definit ion of weak* topology). The continuity of I ( F ) t hus fol lows. 4 An extreme point of a con ve x set F is a point that is not obtainable as a mid-point of two distinct points of F . DRAFT 27 A P P E N D I X I I : P RO O F O F T H E O R E M 1 ( D I S C R E T E C A PAC I T Y - A C H I E V I N G D I S T R I B U T I O N ) The proof i s along the same l ines as W itsenh ausen’ s proo f i n [4], except t hat we have an additional average power constraint on the input. Pr oof: Let S be t he set of all average power constrained di stributions with support in the interval [ A 1 , A 2 ] . The required capacity , by definition, is C = sup S I ( X ; Y ) , where I ( X ; Y ) denotes the mutu al information between X and Y . The achiev ability of the capacity is guaranteed by Theorem 3(a) in Appendix I. [42, Lemm a 3.1] ensures the weak* compactness of the s et S , while weak* continuity of I ( X ; Y ) is easily proven giv en the assumption that the transi tion functions W i ( x ) are continuous. L et S ∗ be a capacity-achieving i nput distribution. The key idea that we employ is a theorem by D ubins [5], which characterizes extreme points of the intersection of a con ve x set with hyperplanes. W e first give some necessary definitions, and then st ate the theorem. Definitions : • Let E be a vector space over th e field of real num bers, and M be a con vex subset of E . M is said to be linearly bounded ( li nearly closed ) if ev ery line intersects M in a bounded (closed) sub set of the lin e. • Let f : E → R be a linear functio nal (not id entically zero). The set { x ∈ E : f ( x ) = c } defines a hy perplane, for any real c . Dubins’ Theor em : Let M be a linearly closed and linearly bo unded con ve x set and U be the intersection of M with n hyperplanes, then every extreme point of U is a conv ex combinati on of at most n + 1 extreme points of M . T o apply Dubins’ theorem to our probl em, w e begin by defining C [ A 1 , A 2 ] : the real normed linear space of all contin uous functions on the interval [ A 1 , A 2 ] , wit h sup-no rm. The dual of C [ A 1 , A 2 ] is the space of functio ns of bounded variations [40, Sec 5.5], and it i ncludes th e (con ve x) set of all distribution functions with support i n [ A 1 , A 2 ] . W e t ake E to be the dual of C [ A 1 , A 2 ] , and M to be the subset o f E consisting of all dis tribution functio ns wit h sup port in [ A 1 , A 2 ] . Note th at the opti mal i nput distri bution S ∗ ∈ M . Let the probability vector of the output Y , when the input is S ∗ , be R ∗ = { p 1 ∗ , p 2 ∗ , · , p K ∗ } . Also, let the avera ge power of the input under th e d istribution S ∗ be P 0 , where P 0 ≤ P . DRAFT 28 Now , consider the foll owing sub set U of M U = { M ∈ M| R ( y ; M ) = R ∗ and E ( X 2 ) = P 0 } . (12) The set U is the intersection of the set M with th e following K hyperplanes H i : Z A 2 A 1 W i ( x ) dM ( x ) = p i ∗ 1 ≤ i ≤ K − 1 (13) and, H K : Z A 2 A 1 x 2 dM ( x ) = P 0 (14) where W i ( x ) are t he transition probability funct ions. Note t hat there are on ly K − 1 hyperplanes in (13 ) s ince the probabil ities mus t sum to 1 , thus making the requi rement on p K ∗ redundant. W e k now t hat the set M is compact in the weak* topology [42, Lemma 3.1]. Also, each of the hyperplanes H i , 1 ≤ i ≤ K − 1 , is a clo sed set sin ce t he functions W i ( x ) are continuous. The hyperplane H K is closed as well , si nce x 2 is a continuous function. Therefore, the set U , being the i ntersection of a weak* com pact set with K cl osed sets, is weak* compact. It i s easy to s ee th at U is a con ve x set as well. On the set U , we have I ( X ; Y ) = H ( Y ) − H ( Y | X ) = − K X i =1 p i ∗ log p i ∗ + Z A 2 A 1 dM ( x ) K X i =1 W i ( x ) log W i ( x ) . As a function o f the dist ribution M ( · ) , we get I ( X ; Y ) = cons tant + l inear , and the li near part is weak* continuou s since K X i =1 W i ( x ) log W i ( x ) is in C [ A 1 , A 2 ] . It follows from Theorem 3 (b) in Appendix I that the continuous li near functional I ( X ; Y ) attains it s maximu m over the compact con vex set U at an extreme po int of U . Howe ver , si nce S ∗ ∈ U , any maxim a over U is a maxima over S as well. Hence, the required capacity is achie ved at an extreme po int o f U . W e now apply Dubin s’ theorem to characterize the extreme poi nts of U . Since U is t he intersection of M wit h K hy perplanes, every extreme point of U is a con vex combination of at most K + 1 extreme points of M . The extreme poi nts of M howe ver are di stributions concentrated at si ngle points with in the interval [ A 1 , A 2 ] . Therefore, w e get that the required capacity is achiev able by a discrete dist ribution w ith at most K + 1 points of suppo rt. DRAFT 29 10 −1 10 0 10 1 10 2 10 −25 10 −20 10 −15 10 −10 10 −5 10 0 y Fig. 8. The second deriv ati ve of h ( Q ( √ y )) is positive ev erywhere. A P P E N D I X I I I : C O N V E X I T Y O F T H E F U N C T I O N h ( Q ( √ y )) T o show con vexity , we verify that the second deriva tive of t he function h ( Q ( √ y )) i s po sitive e verywhere. For y > 2 , we do this analyticall y , while for 0 ≤ y ≤ 2 , th e positivity o f the second deriv ati ve i s demonstrated num erically in Figure 8. Let u ( y ) = h ( Q ( √ y )) . Then, u ′ ( y ) = − e − y / 2 2 √ 2 π y ln 2 ln  1 − Q ( √ y ) Q ( √ y )  Note that 1 − Q ( √ y ) Q ( √ y ) ≥ 1 , ∀ y ≥ 0 . Therefore, to show th at the second deriv ativ e u ′′ ( y ) is positive, it s uf fices t o show that the function v ( y ) = e − y / 2 ln h 1 − Q ( √ y ) Q ( √ y ) i is a d ecreasing function of y . T aking the deriv ative of v ( y ) , we get v ′ ( y ) = − e − y / 2 2  ln  1 − Q ( √ y ) Q ( √ y )  − e − y / 2 √ 2 π y Q ( √ y )(1 − Q ( √ y ))  T o show that v ( y ) is decreasing, it suffic es to show that ln  1 − Q ( √ y ) Q ( √ y )  ≥ e − y / 2 √ 2 π y Q ( √ y )(1 − Q ( √ y )) (15) Using the fact [28, pp. 78] that Q ( y ) ≥ (1 − 1 y 2 ) e − y 2 / 2 y √ 2 π , we get that if y > 1 , then the following condition i s suffi cient for (15) to be true ln  1 − Q ( √ y ) Q ( √ y )  ≥ 1 (1 − 1 y )(1 − Q ( √ y )) (16) DRAFT 30 or , equivalently (1 − 1 y )(1 − Q ( √ y )) ln  1 − Q ( √ y ) Q ( √ y )  ≥ 1 (17) The l eft hand side of (17) is a monoton e increasing function of y . For y = 2 , it equals 1 . 133 . Thus (17) hold s ∀ y > 2 , and hence the second deriv ativ e of h ( Q ( √ y )) mu st be posit iv e for y > 2 . A C K N O W L E D G M E N T The first auth or would like to thank Prof. Janos England er (Department of Probabil ity and Statistics, UCSB), Prof. Shi v Chandrasekaran (Department of Electrical and Computer E ngineer- ing, UCSB) and Mr . V i nay Melkote (Signal Comp ression Laboratory , UCSB) for sev eral useful discussions . R E F E R E N C E S [1] R. Hiremane, Fr om Moor es Law to Intel Innovation-Pr ediction to Reality , T echn ology@Intel Magazine, April 2005. [2] R. W alden, A nalo g-to-Digital Con verter Survey and Analysis , IEEE J. S elect. Areas Comm., 17(4):539–5 50, April 1999. [3] R. G. Gallager, Information Theory and Reliable Communication , John Wile y and Sons, New Y ork, 1968. [4] H.S. Witsenhause n, Some Aspects of Con vex ity Useful in Information Theory , IEEE Tran . Info. Theory , V ol. 26, No. 3, pp. 265-271, May 1980 [5] L.E. Dubins, On Extr eme P oints of Con vex Sets , J. Math. Anal. A ppl., V ol. 5, pp. 237-244, May 1962 [6] N. Phamdo and F . Alajaji, Soft-Decision Demodulation D esign for CO VQ ov er White, Color ed, and ISI Gaussian Channels , IEEE Tran. Comm., V ol. 48, No. 9, pp. 1499-1506, September 2000. [7] F . Behnamf ar , F . Alajaji and T . Linder , Channel-Optimized Quantization W ith Soft-Decision Demodulation for Space-T i me Orthogo nal Block-Coded Channels , IEE E T ran. S ig. Proc., V ol. 54, No. 10, pp. 3935-3946, October 2006. [8] J. A. Nossek and M. T . Ivrlac, Capacity and Coding for Quantized MIMO Systems , In Proc. IWCMC, 2006. [9] O. Dabeer , J. Si ngh and U. Madho w , On the Limits of Communication P erformanc e wi th One-Bit A nalog -T o-Digital Con version , I n Proc. S P A WC’2006, Cannes, F rance. [10] J. Si ngh, O. Dabeer and U. Madhow , Communication Limits with Low Pr ecision Analog-T o-Digital Con version at the Receiver , In Proc. ICC’2007, Glasgow , S cotland. [11] D. P orcino and W . Hurt Ult ra-W ideband Radio T echno logy: P otential and Challenges Ahead , IEEE Communications Magazine, V ol. 41, No. 7, pp. 66-74, July 2003 [12] P . Smulders, Exploiting the 60 GHz Band f or Local W ir eless Multimedia Access: Pr ospects and Futur e Dir ections , IEEE Communications Magazine, V ol. 40, No. 1, pp. 140-147, January 2002 [13] R. Blazquez, F . S. Lee, D. W entzlof f, P . Ne waskar , J. Powell, and A . C handraka san, Digital Arc hitectur e for an Ultra- W ideban d Radio Receiver , In Proc. 58th I EEE V ehicular T echnology Conference, 2003. [14] I. D. O Donnell and R. W . Brodersen, A n Ultra-W ideband T ransceiver Ar chitectur e for Low P ower , Low Rate, W ireless Systems , Proc. 19th International Conference on VLS I Design, V ol. 54, September 2005. DRAFT 31 [15] S . Hoyos, B. M. Sadler , and G. R. Arce, Monobit Digital Receivers for Ultrawideban d , , IEEE Trans actions On Wireless Communications, V ol. 5, July 2005. [16] I. S. -C. Lu, N. W este, and S. Parameswa ran, Adc Pre cision Requir ement for Digital Ultrawideband Rece ivers with Sublinear F r ont-Ends: A P ower and P erformance perspe ctive , In Proc. 19th International Conference on VLSI Design (V LSID06), 2006. [17] S . Ho yos and B. M. Sadler , Fr equency-Domain Implementation of t he T ransmitted-Refer ence Ult ra-W ideband Receiver , IEEE Transac tions On Micro wav e Theory And T echniques , V ol. 54, April 2006. [18] W . Namgoong, A Channelized Digital Ult rawideba nd Receiver , IEEE T ransactions On W ireless Communications, V ol. 2, May 2003. [19] O. Dabeer and U. Madho w , Detection and Interference Suppr ession for Ultrawideband Signaling with Analog Pr ocessing and One Bit A/D , In Proc. Asilomar Conference on Signals, S ystems, and Compu ters, Pacific Grov e, CA, Nov ember 2003. [20] E . Masry T he Reconstruction of A nalog Signals from the Sign of Their Noisy Samples , IEEE Transaction s on Information Theory , V ol. 27, pp. 735-745, Novemb er 1981. [21] Z . Cvetk ovic and I. Daubechies, Single-bit Oversampled A/D Con version wi th E xponential Accuracy in the Bi t-Rate , In Proceedings DC C, pp. 343352, Utah, USA, March 2000. [22] P . Ishw ar , A. Ku mar and K. Ramachandran , Distributed Sampling for De nse Sensor Networks: A Bit-Conserva tion Principal , In P roceedings IPSN, pp. 17-31, Palo Alto, USA, 2003. [23] A. Host-Madsen and P . Handel, Effects of Sampling and Quantization on Single-T one Fr equency E stimation , IEE E T ransactions on S ignal Processing, V ol. 48, pp. 650662, March 2000. [24] T . Andersson, M. Skoglund, and P . Handel, F r equency Estimati on by 1-bit Quantization and T able Look-Up Proces sing , In P roceedings of European Signal P rocessing Conference, S eptember 2000. [25] D. Rousseau, G. V . Anand, and F . Chapeau-Blonde au, Nonlinear Estimation fr om Quantized Signals: Quantizer Optimization and Stochastic Resonance , In Third International Symposium on Physics in Signal and Image Processing, (Grenoble, Fr ance), pp. 89-92, January 2003. [26] O. Dabeer and A. Karnik, Signal P ara meter Estimation Using 1-bit Dither ed Quantization , IEEE T ran. Info. Theory , December 2006. [27] O. Dabeer and E. Masry , Multivariate Signal P arameter Estimation Und er Depende nt Noise Fr om 1-bit Dither ed Quantized Data , IEEE Tran . Info. Theory , T o Appear , April 2008. [28] U. Madho w , Fundamentals of Digital Communications , Cambridge Univ ersity Press, 2008. [29] J. G. Smith, On the I nformation Capacity of P eak and Av erag e P ower Constra ined Gaussian Channels , Ph.D. Dissertation, Univ . of California, Berkeley , December 1969. [30] I. C. Abou-F aycal, M. D. T rott and S. Shamai, T he Capacity of Discr ete-T ime Memoryless Rayleigh F ading Channels , IEEE Tran. Info. Theory , V ol. 47, No. 4, pp. 1290-1301 , May 2001. [31] J. H uang and S. P . Meyn, Characterization and Computation of Optimal Distribution s for Channel Coding , IEEE T ran. Info. T heory , V ol. 51, No. 7, pp. 2336-2351, July 2005. [32] T . Cover and J. Thomas, Elements of Information Theory , Wile y Series i n T elecommu nications, New Y ork, 2nd Edition. [33] I. Csiszar and J. Korner , Information Theory : Coding Theor ems F or Discr ete Memoryless Systems , Acade mic Press, 1981. [34] M. Chiang and S. B oyd , Geometric Pro gramming Duals of Channel Capacity and R ate Distortion , IE EE Tran. Info. Theory , V ol. 50, No. 2, pp. 245-258, February 2004. DRAFT 32 [35] M. Katz and S. Shamai, On the Capacity-Ach ieving Distribution of the Discre te-T ime Noncoher ent and P artially Cohere nt A WGN channels , I EEE Tran . Info. Theory , V ol. 50, No. 10, pp. 2257-2270, October 2004. [36] A. L apidoth and S . M. Moser , Capacity Bounds via Duality with Applications to Multiple-Antenna Systems on Flat F ading Channels , IEEE Trans. Inform. Theory , V ol. 49, No. 10, pp. 2426-2467, October 2003 [37] S . Boyd and L. V andenberghe, Conve x Optimization , Cambridge Univ . Press, 2004. [38] C. H. Doan, S . Emami, A. M. Niknejad and R. W . Brodersen, Millimeter-wave CMOS design , IEEE J. S olid-State Ci rcuits, V ol. 40, No. 1, pp. 144-155, January 2005. [39] M.S . Braasch, A.J. van Dierendonck GPS Receiver A r chitectur es and Measur ements , Proceedings of the IE EE, V ol. 87, No. 1, pp. 48-64, January 1999. [40] D. G. Luenberger , Optim ization By V ector Space Methods , John Wiley & S ons, Inc., 1969. [41] C. Niculescu and L. E. Persson, Con vex Functions and T heir Applications , Springer , 2005. [42] M. Fozunbal, S. W . McLaughlin and R. W . Schafer , Capacity Analysis for Continuous-Alphabet Channels W ith Side Information, P art I: A General F rame work , IEEE Tran. Info. Theory , V ol. 51, No. 9, pp. 3075-3085, 2005. [43] C. I. Chang and L. D. Davisson, On Calculating the Capacity of an Infinite-Input Finite(Infin ite)-Output Channel , IEEE T ran. Info. Theory , V ol. 34, No. 5, pp. 1004-1010, September 1988. DRAFT