Shannon Meets Carnot: Mutual Information Via Thermodynamics

1 Shannon Meets Carnot: Mutual Information V ia Thermodynamics Ori Shental and Ido Kanter Abstract In this contribution, the Gaussian channel is represen ted as an eq uiv alent thermal system allowing to express its input-o utput mutu al in formatio n in terms of thermod ynamic qu antities. This thermod ynamic description o f the mutual info rmation is based up on a generalization of the 2 nd thermod ynamic law an d provides an alternativ e proof to the Guo-Shamai-V erd ´ u theorem, gi ving an intrig uing connection between this remark able th eorem and the most fundam ental laws of nature - the laws of thermo dynamics. Index T e rms: Thermodynami cs, mutual information, Gaussian channel, Guo-Shamai-V e rd ´ u theorem, minimum mean-square error . I . I N T RO D U C T I O N The laws of thermodynamics describe the transp ort of h eat and work in macroscopic ( i.e. , lar ge scale) processes and play a fundamental role in the physical sciences. The theory of thermodynamics is primarily an i ntellectual achievement of the 19 th century . The first analysis of heat engi nes was given by the French engineer Sadi Carnot in his seminal 1824 publ ication (‘ Reflections on the Mot ive P ower of F ir e and on Machines F itted to De velop t hat P ower ’ [1], [2]), laying the foundations to th e 2 nd law o f thermodynam ics. This paper marks t he start of thermodynamics as a modern science [3]. O. Shental i s with the Center for Magnetic Recording Research (CMRR), Uni versity of California, San Diego (UCSD), 9500 Gilman Driv e, La Jolla, CA 92093, US A ( e-mail: oshental@ucsd.edu). I. Kanter is with the Minerva Center and Department of Physics, Bar-Ilan Uni versity , Ramat-Gan 52 900, Israel (e-mail: kanter@mail.biu.ac.il). DRAFT 2 The classical th eory of thermodynamics was formulated in con sistent form by gi ants like Joule, Clausius, Lord Kelvin, Gibbs and others. The atomic, or m icroscopic ( i.e. , s mall scale) approach to statistical thermo dynamics was mainly de veloped through the pioneering work o f Clausius, Maxwell , Boltzmann and Gibbs, laying t he foundations to the more g eneral di scipline of statisti cal mechanics [4]. In particular , th e 2 nd thermodynamic l aw , for quasi-stati c processes, linearly relates the change in t he entropy , dS , to the amoun t of heat, d ¯ Q , abso rbed to a system at equilibriu m, d ¯ Q = T dS , where T is the temperature of the system. Ho wev er , t he 2 nd law , in its classical formulation, is suited only for systems with ener gy Hamiltonian (function), E , which is not an e xplicit function of the temp erature and fails to capture the p hysical behavior of systems with temperature-dependent ener gy levels. Whil e such temperature-dependent energy function is uncommon in the study of natural and artificial systems in ph ysics, it surprisingly does arise in mo deling communication channels, like the popular Gaussi an-noise commu nication channel [5]–[7], as a (quasi-static) thermal system [8]–[10]. In this cont ribution, we generalize t he 2 nd thermodynamic law to encompass sys tems with temperature-dependent Hamiltonian and obtain the generalized la w d ¯ Q = T dS + < d E /dT > d T , where < · > denot es aver aging over the s tandard Boltzmann distribution. Consequently , it allows for an alternati ve physical d escription to the Shannon-theoretic notions of i nformation entropy and m utual information [11] v ia the thermodynamic quanti ties of ener gy and temperature. As an example, the correct expressions for t he mutual information of a Gauss ian channel with Bernoulli-1/2 i nput and Gaussian input are computed from the th ermodynamic representati on, where the latter corresponds to the Shannon capacity . Guo, Shamai and V erd ´ u [12] ha ve recently re vealed a simple, yet po werful relationsh ip (hereinafter termed GSV theorem) between inform ation and estim ation theories. This cross- theory theorem bridges ov er the no tions of Shannon’ s mut ual information and mini mum mean- squre error (M MSE) for t he comm on additive white Gaussian noise channel. Based on the thermodynamic expression of mutual informati on, the GSV theorem is natu rally re-deriv ed. This directly li nks t he GSV theorem to t he m ost profound laws o f nat ure and gives the phy sical origi n to this remarkable formula. The paper i s organized as foll ows. The thermodynamic background is discussed in Sections II and III . In Section IV t he Gaussi an channel is represented throug h an equiv alent thermal system DRAFT 3 giving a thermodynamic e xpression to the notions of information entropy a nd mutual information. In S ection V the GS V theorem is pro ven via thermodynamics. W e conclu de the paper in Section VI. W e s hall use th e following n otations. P ( · ) is us ed to denot e either a probability mass functi on (pmf), Pr( · ) , or a p robability density function (pdf), p ( · ) , depending on the random v ariable having either discrete or continuous support, respecti vely . Random va riables are denoted by upper case letters and their values denoted by lower case letters. The symbol E X {·} denot es expectation of the random object withi n the brackets with respect to th e subscript random v ariable. The natural logarithm , log , is used. The support of a v ariable X is denot ed by X . I I . C L A S S I C A L T H E R M O DY N A M I C S In this s ection we con cisely summarize the fund amental results of classi cal thermodynamics, essential to our discussion. The interested reader is strongly encouraged to find a thorough introduction to thermodynamics in one of num erous textbooks ( e.g. , [13]). A thermodynam ic system is defined as the part o f the universe under con sideration, separated from the rest of the uni verse, referred to as en vironment, surroundings or reservoir , b y real or imaginary boundary . A non-is olated t hermodynamic s ystem can exchange energy in the form of heat or work with any other system. Heat is a p rocess by which energy is added t o a system from a high-temp erature source, or lost to a l ow- temperature sink. W ork refers to forms of ener gy transfer which can be accounted for i n terms of changes i n the macroscopic physical variables of t he sys tem ( e.g. , volume or pressu re). For example, energy which goes into expanding the volume o f a system again st an e xternal pressure, by driving a pist on-head out of a cylinder against an e xternal force. Hereinafter we consi der a t hermal system which does not perform work, mechanical or ot her . This thermal system is assumed to be in equ ilibrium, that is all the descriptive macroscopic p arameters of the sys tem are ti me-independent. W e also assum e that the process of exchanging heat is infinitesim ally quasi-static, i.e. , it is carried out sl owly eno ugh that the system remains arbitrarily close t o equilibrium at all stages of the process. The underlyi ng laws of thermodyn amics consist o f purely macroscopi c statements which m ake no reference to the microscopi c properties of the sys tem, i.e. , to the molecules or particles of which they consist . In the following statem ents we present the thermodynam ic laws i n a form DRAFT 4 rele vant to the thermal system under consideration. 1 Laws of the rmodynamics. 1 st Law . (conservation of ener gy) A s ystem in equilibr ium can be characterized by a quantity U , called the ‘internal ener gy’. If the system is not isolated, interact with an other system and no work is done by it, t he r esulting change in the internal ener gy can be writ ten in the form dU = d ¯ Q, (1) wher e d ¯ Q is the amoun t of heat absorbed by the system. 2 2 nd Law . (definition of temperatur e) A system in equilibrium can be characterized b y a quant ity S , called the ‘thermodyna mic ent r opy’. If the syst em i s not isolated and under go es a qu asi-static infinitesimal pr ocess in which i t absorbs heat d ¯ Q , then dS = d ¯ Q T , (2) wher e T is a quantity characteristic of the system and is called the ‘absolute temperatur e’ of the system. 3 rd Law . (zer o e ntr opy) The thermodynamic entr opy S of a s ystem has a limiting property that as T → 0 + , S → S 0 , (3) wher e S 0 is a cons tant (usually 0 in case of non-de generate ground state ener gy) independent of all parameters of the particular system. Incorporating the three laws of thermodynamics together , a combined law describing the thermodynamic entropy as an integration function ov er the temperature is obtained S ( T ) = Z T 0 1 γ dU ( γ ) = Z T 0 1 γ dU ( γ ) dγ dγ = Z T 0 C V ( γ ) γ dγ , (4) where C V ( T ) , dU ( T ) dT (5) 1 For concisen ess the 0 th law’ s statement is omitted. 2 The infinitesimal heat is den oted by d ¯ rath er tha n d because, in mathematical t erms, it is not an exact differential. The integral of an ine xact dif ferential depen ds upon the p articular path taken through the space of (thermodynamic) parameters while the integral of an exact dif ferential depends only upon the i nitial and final states. DRAFT 5 is known as the heat capacity (at constant volume V ). The heat capacity is a n on-negati ve temperature-dependent measurable quantity describi ng the amount of heat required to change the system’ s temperature by an infinitesi mal degre e. Let us now defi ne the in verse tem perature β , ( k B T ) − 1 , where the constant k B is the Boltz- mann constant. In t his contribution we arbitrarily set k B = 1 , thus β = 1 /T . Hence, the ent ropy integral (4) can be re written in terms of the in verse temperature as S ( β ) = − Z ∞ β γ C V ( γ ) dγ , (6) where C V ( β ) , dU ( β ) dβ = − T 2 C V ( T ) . (7) I I I . S T A T I S T I C A L T H E R M O DY N A M I C S Moving to the microscopic perspectiv e of statistical thermodyn amics [13], [14], the probability , P ( X = x ) , of finding the system in any one particular microstate, X = x , of energy level E ( X = x ) is dictated according to the canoni cal Boltzmann distribution [4] P ( X = x ) = 1 Z exp  − β E ( X = x )  , (8) where Z , X x ∈X exp  − β E ( X = x )  (9) is the partit ion (normali zation) function, and the sum extends ove r all pos sible m icrostates of the system. Applying the Boltzmann distribution, the m acroscopic quant ities of internal ener g y and en- tropy can be described m icroscopically as t he average energy and the a verage {− log P ( X ) } , respectiv ely . Explicitl y , U = E X {E ( X ) } , (10) S = E X {− log P ( X ) } , (11) and it can be easily verified that the following relation holds log Z = − β U + S. (12) DRAFT 6 I V . T H E G AU S S I A N C H A N N E L A S A T H E R M A L S Y S T E M Consider a real-valued channel wit h input and out put random variables X and Y , respecti vely , of the form Y = X + N , (13) where N ∼ N (0 , 1 / snr ) is a Gaussian noise independent of X , and snr ≥ 0 is t he channel’ s signal-to-noise rati o (SNR). The input is taken from a probabil ity di stribution P ( X ) that satisfies E X { X 2 } < ∞ . 3 The Gaussi an channel (13) (and any other communication channel) can be also viewed as a physical system, op erating under the l aws of thermodynam ics. The microstates of the thermal system are equivalent to the hidden values of th e channel’ s inpu t X . A comparison of the channel’ s a-posteriori probability distribution, gi ven b y P ( X = x | Y = y ) = P ( X = x ) p ( Y = y | X = x ) p ( Y = y ) (14) = √ snr P ( X = x ) √ 2 π p ( Y = y ) exp  − snr 2 ( y − x ) 2  (15) = exp  − snr ( − xy + x 2 2 − log P ( X = x ) snr )  P x ∈X exp  − snr ( − xy + x 2 2 − log P ( X = x ) snr )  , (16) with the Boltzmann dist ribution law (8) yi elds the following mapping of the in verse temperature and energy of the equi valent thermal system snr → β , (17) − xy + x 2 2 − log P ( X = x ) β → E ( X = x | Y = y ; β ) . (18) Note t hat t he temperature ( i.e. , the noise v ariance according to the m apping (17)) can be increased gradually from the absolute zero to its target value T = 1 / β in infinitesimally sm all steps, thus the Gaussian channel system can remain arbitrarily close to equili brium at all stages of this process. Hence, the equivalent thermal sys tem exhibits a q uasi-static i nfinitesimal process. Interestingly , the notion of quas i-statics is reminis cent t o t he concept of Gaussian pipe in the SNR-incremental Gaussian channel approach t aken by Guo et al. [12, Section II-C]. Thus, we 3 Similarly to the formulation in [12] for E X { X 2 } = 1 , snr follo ws the usual notion of SNR. For E X { X 2 } 6 = 1 , snr can be regarded as the gain i n the output SNR due to the channel. DRAFT 7 can apply the entropy i ntegral (6) obtai ned from thermo dynamics to the thermal system being equiv alent to the Gaussian channel, yi elding S ( β ) = S ( X | Y = y ; β ) = − Z ∞ β γ C V ( γ ) dγ = − Z ∞ β γ dU ( Y = y ; γ ) dγ dγ , (19) where following (10) the internal ener gy , U ( Y = y ; β ) = E X | Y {E ( X | Y = y ) } , is the energy a veraged over all possi ble v alues of X , give n y . The pos terior informatio n (Shannon) entropy , H ( X | Y ; β ) , (in nats) of the channel can be expressed via the thermodynamic entropy conditioned on Y = y , S ( X | Y = y ; β ) (19), as H ( X | Y ; β ) = E Y { S ( X | Y = y ; β ) } = − E Y ( Z ∞ β γ dU ( Y ; γ ) dγ dγ ) . (20) The input’ s entropy can also be reformulated in a si milar manner , since H ( X ) = H ( X | Y ; β = 0 ). Hence, H ( X ) = − E Y ( Z ∞ 0 γ dU ( Y ; γ ) dγ dγ ) . (21) Now , the input-out put mutual information can be described via t hermodynamic qu antities, namely the energy , E , and in verse temperature, β , as I ( X ; Y ) = I ( β ) , H ( X ) − H ( X | Y ; β ) (22) = − E Y ( Z β 0 γ dU ( Y ; γ ) dγ dγ ) (23) = −  γ E Y { U ( Y ; γ ) }  β 0 + E Y ( Z β 0 U ( Y ; γ ) dγ ) , (24) where (24) is obtained using integration by parts. Note that this thermodynamic interpretation to th e mutual information holds not only for th e Gaussian channel, b ut also for an y channel which can be described by a thermal system exhibiting quasi-static heat transfer . In the foll owing we illustrate the utili zation of (24) by re-deriving the correct expression for the mutual information of a Gaussian channel with Bernoulli-1/2 inp ut. Example: Ga ussian Chan nel with Bernoulli- 1 / 2 Input Since t he inpu t X in this case i s bi nary and equiprobable, i.e. P ( X = 1) = P ( X = − 1) = 1 / 2 , the X 2 / 2 = 1 / 2 and log P ( X = x ) /β t erms of the Gaussian channel’ s energy (18) are DRAFT 8 independent of X and can be dropped 4 , leaving us with the expression, in dependent of β , E ( X = x | Y = y ) = − xy . (25) The a-posteriori probabilit y mass function gets the form Pr( X = x | Y = y ) = exp ( xβ y ) exp ( β y ) + exp ( − β y ) , x ∈ ± 1 . (26) Hence, the internal energy is U ( Y = y ; β ) = E X | Y {E ( X | Y = y ) } = − y ta nh( β y ) . (27) The marginal pdf of the output is then giv en by p ( Y = y ) = √ β 2 √ 2 π exp  − β 2 ( y − 1) 2  + exp  − β 2 ( y + 1) 2  ! . (28) Thus, − β E Y { U ( Y ; β ) } = − Z ∞ −∞ y t a nh( β y ) p ( Y = y ) d y ( 29) = − β 2 √ 2 π Z ∞ −∞ y exp  − β 2 ( y − 1) 2  − exp  − β 2 ( y + 1) 2  ! dy (30) = β 2 (1 − ( − 1)) = β (31) and E Y ( Z β 0 U ( Y ; γ ) dγ ) = Z ∞ −∞ p ( Y = y ) Z β 0 U ( Y = y ; γ ) d γ dy (32) = − 1 √ 2 π Z ∞ −∞ exp  − y 2 2  log cosh ( β − p β y ) dy (33) giving, bas ed on (24), I ( β ) = β − 1 √ 2 π Z ∞ −∞ exp  − y 2 2  log cosh ( β − p β y ) dy , (34) which is identical to the known Shanno n-theoretic result (see, e.g. , [12, eq. (18)] and [15, p. 274]). 4 Cancelled out by the same terms fr om the denomin ator in (16). DRAFT 9 A. Generalized 2 nd Law When trying t o repeat th is exercise for a Gaussian i nput one finds that (24) fails to reproduce I ( snr ) = 1 / 2 log (1 + snr ) , which is the ce lebrated formul a for the Shannon capacity o f th e Gaussian channel. This observation can be explained as follows. The 2 nd thermodynamic law , as stated above, holds only for sys tems with an ener gy functi on, E , which is not a fun ction of the temp erature. While the temperature-independence of E is a well-kno wn concepti on in the in vestigation of both natural and artificial systems in thermodynamics, and phys ics at large, such independence does not necessarily hold for com munication channels and particularly fo r the Gaussian channel. Actually , one can easily observe that the Gaussian channel’ s ener gy (18) is indeed an explicit functio n of the temperature v ia th e log P ( X = x ) /β term, unless this term can be dropp ed by abs orbing it into t he partition function, as happens for equiprobable input dis tributions, like the discrete Bernoulli -1/2 or the continuous uni form distributions. This is exactly the reason why (24) do es su cceed to compute correctly t he m utual information for the particular case of a Bernoulli - 1 / 2 input source. In order t o capture the temperature-dependent nature of the energy in s ystems like t he commu - nication channel, we generalize t he formul ation of the 2 nd law of thermodynamics dS = d ¯ Q/T . Generalized 2 nd Law . (r edefinition of temperatur e) If t he t hermal system is n ot isol ated and under go es a quasi-stati c i nfinitesimal pr ocess in which it absorbs heat d ¯ Q , then dS = d ¯ Q T − 1 T E X  d E ( X ) dT  dT . (35) Pr oof : The diff erential of the partition function ’ s logarithm , log Z , can be written as d log Z = d log Z dβ dβ . (36) Utilizing the identit y (12), one obtains dS = d ( β U ) + d log Z dβ dβ . (37) Since for T -dependent ener gy d log Z dβ = − U − β E X  d E ( X ) dβ  , (38) we get dS = d ( β U ) − U d β − β E X  d E ( X ) dβ  dβ = β dU − β E X  d E ( X ) dβ  dβ , (39) DRAFT 10 where the r .h.s. results from Leibni z’ s law (product rule). Recalling that according to the 1 st law dU = d ¯ Q concludes the proo f. The generalized 2 nd law of thermodynamics (35) has a clear p hysical interpretation. For simplicit y , let us assume that the examined system i s characterized by a comb of discrete energy lev els E 1 , E 2 , . . . . The heat absorbed into the T -dependent system has the following dual effect: A first contribution of the heat, dU − E X { d E ( X ) /d T } dT , increases the temperature of the system while the second contribution, E X { d E ( X ) /d T } dT , goes for shifting the energy comb . Howe ver , the shift of the energy comb does not affect the entropy , since t he occupation o f each ener gy lev el r emains the same, and the entropy is independent o f the energy v alues which stand behind the labels E 1 , E 2 , . . . . The change in t he entropy can be done only by moving part of the occupation o f one to oth of the energy comb t o the neighboring t eeth, and this can be achie ved only b y changin g the temp erature. Hence, the effecti ve heat contri buting to the ent ropy is d ¯ Q − E X { d E ( X ) /d T } dT , and this is the physical explanation to the generalized 2 nd law (35). Note that for T -independent energy , the cl assical 2 nd law (2) is immediately obtained. Note also, that the 1 st law remains unaffected, dU = d ¯ Q , since both w ays of heat flo w absorption into the syst em are ev entually contributing to the av erage int ernal energy U . The generalized 2 nd law specifies the trajectory , the weight of each one of th e two possib le heat flo ws, at a given temperature. The temperature, originally defined by the 2 nd law as T = d ¯ Q/dS , is redefined now as 1 T = dS/d ¯ Q 1 − E X { d E ( X ) /dT } d ¯ Q/dT = dS/d ¯ Q 1 − E X { d E ( X ) /dT } C V ( T ) . (40) This redefinition has a m ore com plex form and in volves an implicit functi on of T s ince t he temperature appears on both sides of the equati on. Based on the generalized 2 nd law , the thermodynamic e xpression for the mutual informa- tion (22) of quasi-st atic ( e.g. , Gaussian) communicati on channels can be reformulated as I ( X ; Y ) = − E Y ( Z β 0 γ  dU ( Y ; γ ) + E X | Y  d E ( X | Y ; γ ) dγ  dγ  ) (41) = − E Y ( Z β 0 γ  dU ( Y ; γ ) dγ + E X | Y  d E ( X | Y ; γ ) dγ   dγ ) (42) = −  γ E Y { U ( Y ; γ ) }  β 0 + E Y ( Z β 0  U ( Y ; γ ) + γ E X | Y  d E ( X | Y ; γ ) dγ   dγ ) . (43) DRAFT 11 Again, the example of a Gaussian channel with, thi s time, standard G aussian inpu t is used to illustrate the utilization of (43) for th e correct deriv ation of the mutual informati on. Example: Ga ussian Chan nel with N (0 , 1) Input Since in this case log ( P ( X )) /β = − x 2 / (2 β ) , t he energy (18) of the Gaussian channel system becomes an explicit function of β , give n by E ( X = x | Y = y ; β ) = − xy + x 2 2  1 + β β  , (44) and the deriv ative of th is function with respect to β yields ∂ E ( X = x | Y = y ; β ) ∂ β = − x 2 2 β 2 . (45) The a-posteriori probabilit y density function is p ( X = x | Y = y ; β ) = N  β y 1 + β , 1 1 + β  . (46) Hence, the internal energy is U ( Y = y ; β ) = E X | Y {E ( X | Y = y ; β ) } = − y 2 β 2(1 + β ) + 1 2 β , (47) and the deriv ative of th e ener gy a veraged over all possible inputs is E X | Y ( ∂ E ( X | Y = y ; β ) ∂ β ) = − 1 2 β 2  1 1 + β + β 2 y 2 (1 + β ) 2  . (48) The marginal pdf of the output is giv en by p ( Y = y ) = N 0 , 1 + β β ! . (49) Thus, −  γ E Y { U ( Y ; γ ) }  β 0 = β 2 − 1 2 −  − 1 2  = β 2 , (50) DRAFT 12 and E Y ( Z β 0  U ( Y ; γ ) + γ E X | Y  d E ( X | Y ; γ ) dγ   dγ ) (51) = E Y (  − 1 + γ 2 + 1 + γ 2 γ log(1 + γ ) + 1 2 log(1 + γ ) − Y 2 2 log(1 + γ ) − Y 2 2(1 + γ )  β 0 ) (52) = E Y ( − β 2 + 1 + β 2 β log(1 + β ) − 1 2 + 1 2 log(1 + β ) − Y 2 2 log(1 + β ) − Y 2 2(1 + β ) + Y 2 2 ) (53) = − β 2 + 1 2 log (1 + β ) (54) giving, bas ed on (43), I ( X ; Y ) = 1 2 log (1 + β ) (55) and the Shannon capacity [11] is derived fr om the perspective of th ermodynamics. V . G U O - S H A M A I - V E R D ´ U T H E O R E M In this s ection we prov e the Guo-Shamai-V erd ´ u (GSV) theorem from the 2 nd law of t hermo- dynamics for systems with T -dependent ener gy and the resulting thermodynam ic representation of th e mut ual information. Thus, we show that this fascinating relation between information theory and estimation theory is actually an ev olutio n of the most profound laws o f nature. T o start with, let us restate the GSV theorem. Consider a Gaussian channel of the form Y = √ snr X + N , (56) where N ∼ N (0 , 1) is a s tandard Gaussian noi se i ndependent of X . The mutual information , I ( snr ) (22), and the mini mum mean-square error , defined as mmse ( X | √ snr X + N ) = mmse ( snr ) = E X,Y { ( X − E X | Y { X | Y ; snr } ) 2 } , (57) are both a function of snr and main tain the following relation. GSV Theor em. [12, Theor em 1] F or every i nput distr ibution P ( X ) t hat sa tisfies E X { X 2 } < ∞ , d d snr I ( X ; √ snr X + N ) = 1 2 mmse ( X | √ snr X + N ) . (58) DRAFT 13 Note that the GSV -based expression of the mutual information, I ( X ; √ snr X + N ) = 1 2 Z snr 0 mmse ( X | √ γ X + N ) dγ , (59) resembles it s thermo dynamic expression (42) i n the sense that both are an out come of integration with respect to SNR (or in verse temperature, β ). Hence, the roots of this integration in the GSV theorem m ay be attributed to the 2 nd law of thermodynamics. No te howe ver to th e op posite o rder of i ntegration in t he two expressions , where in the GSV expression (59) t he in ner int egration (within the definitio n of MMSE) is over y and t he outer integration i s over SNR, a nd vi ce versa for the thermodynamic expression (42). Exchanging the o rder o f integration i n the latter (which is not tri vi al since p ( Y = y ) is its elf a function o f β ) yields, as we shall see, an inte grand of d β which is equal to mmse ( β ) / 2 . In the foll owing proof Lemma 1 from [12], which underlines the main proof o f t he GSV theorem in [12], is proven di rectly from the t hermodynamic d escription of the mutual information (41). Pr oof : Adopting the SNR-incremental channel approach [12, Eq. (30)-(41)] and mappin g again snr → β dI ( X ; Y ) dβ = I ( X ; Y 1 ) − I ( X ; Y 2 ) δ = I ( β + δ ) − I ( β ) δ = I ( X ; Y 1 | Y 2 ) δ , (60) where I ( X ; Y 1 | Y 2 ) is the mutu al information of the incremental Gaussian channel Y 1 = √ δ X + N , (61) where N is a stand ard Gaussi an noise, X is taken from the conditional probabil ity P ( X | Y 2 ) , δ → 0 and Y 2 = X + N (0 , 1 /β ) (62) with X ∼ P ( X ) . Hence, we hav e to prove I ( X ; Y 1 | Y 2 ) δ = 1 2 mmse ( β ) . (63) Now , the principl es of t hermodynamics come into action. For this incremental channel (61), the ener gy and i ts deri vati ve are given by E ( X = x | Y 1 = y 1 , Y 2 = y 2 ; δ , β ) = − xy 1 √ δ + x 2 2 − log P ( X = x | Y 2 = y 2 ; β ) δ (64) and d dδ E ( X = x | Y 1 = y 1 , Y 2 = y 2 ; δ , β ) = xy 1 δ − 3 2 2 + log P ( X = x | Y 2 = y 2 ; β ) δ 2 . (65) DRAFT 14 Using the thermody namic e xpression for the m utual information (41) and recalling t hat δ → 0 , one gets I ( X ; Y 1 | Y 2 = y 2 ) = − E Y 1 | Y 2 ( Z δ 0 γ  dU ( Y 1 | Y 2 = y 2 ; γ , β ) + E X | Y 1 ,Y 2  ∂ ∂ γ E ( X | Y 1 , Y 2 = y 2 ; γ , β )  dγ  ) (66) = − E Y 1 | Y 2 ( E X | Y 1 ,Y 2  δ E ( X | Y 1 , Y 2 = y 2 ; δ , β ) + δ 2 d dδ E ( X | Y 1 , Y 2 = y 2 ; δ , β )  ) (67) = − E Y 1 | Y 2 ( − E X | Y 1 ,Y 2 { X | Y 1 , Y 2 = y 2 } Y 1 √ δ 2 + δ E X | Y 1 ,Y 2 { X 2 | Y 1 , Y 2 = y 2 } 2 ) (68) = − E Y 1 | Y 2 ( − √ δ 2 Y 1 E X | Y 2 { X | Y 2 = y 2 } + δ 2 E X | Y 2 { X 2 | Y 2 = y 2 } ) (69) = δ 2  − E 2 X | Y 2 { X | Y 2 = y 2 } + E X | Y 2 { X 2 | Y 2 = y 2 }  (70) = δ 2 E X | Y 2 { ( X − E X | Y 2 { X | Y 2 = y 2 } ) 2 | Y 2 = y 2 } , (71) where (69) is based on the fact that for an infinitesimal snr , δ → 0 , the expectation E X | Y 1 ,Y 2 → E X | Y 2 , while (70) results from the independence of N (61) w ith Y 2 (62) [12, Section II-C]. A veraging over Y 2 on both sides of the equation, we ob tain the desired result I ( X ; Y 1 | Y 2 ) = δ 2 E X,Y 2 { ( X − E X | Y 2 { X | Y 2 } ) 2 } = δ 2 mmse ( β ) . (72) V I . C O N C L U S I O N In this paper , the mutual information of Gaussian channels is described via t hermodynamic terminology . As a b yproduct, an in timate link is rev ealed between the GSV t heorem and the basic laws of thermodynami cs. M ore generally , the revised 2 nd law for thermal sys tems with temperature-dependent energy le vels enables to q uantitatively b ridge between the realm of ther - modynamics and in formation theory for q uasi-static s ystems. It is anti cipated that this su bstantial theoretical conn ection between the foundation laws of thermodynamics and inform ation theory will open a horizon for new discoveries and d e velopment in th e stud y of both artificial and natural systems , to wards a possibl y more syner geti c foundation to these two vital discipli nes. DRAFT 15 R E F E R E N C E S [1] S. C arnot, R ´ eflexion s sur la puissance motrice du feu et sur les machines pro pr es ` a d ´ evelopper cette puissance . Paris: Bachelier , 1824. [2] E. Mendoza, Ed., Reflections on the Motive P ower of Fir e: And Other P apers on the Second Law of Thermodynamics . Ne w Y ork: Dover Publications, 2005. [3] E. Mendoza , “ A sketch for a history of early thermodynamics, ” Physics T oday , p. 32, Feb 1961. [4] L. E. Reichl, A Modern Course in Statistical Physics . Univ ersity of T exas P ress, 1980. [5] T . M. Cov er and J. A. Thomas, Elements of Information Theory . John Wiley and Sons, 1991. [6] J. G. Proakis, Digital Communications , 4th ed. New Y ork, NY : McGraw Hill Higher Education, 2000. [7] R. G. Gallager , Principles of Digital Communication . Cambridge, UK: Cambridge Univ ersity Press, 2008. [8] N. Sourlas, “Spin-glass models as error-correcting codes, ” Natur e , vol. 339, pp. 693–695, jun 1989. [9] P . Ruj ´ an, “Finite temperature error-correcting codes, ” Phys. Rev . Lett. , vol. 70, no. 19, pp. 2968–297 1, May 1993. [10] H. Nishimori, S tatistical Physics of Spin Gl asses and Information Processin g . Oxford, UK: Oxford Unive rsity Press, 2001. [11] C. E. Shannon, “ A mathe matical theory of communication, ” Bell System T echnical Journal , vol. 27, p p. 379– 423 and 623–65 6, Jul. and Oct. 1948. [12] D. Guo, S. Shamai (S hitz), and S. V erd ´ u, “Mutual information and minimum mean-square error in Gaussian channels, ” IEEE T rans. Inform. Theory , vol. 51, no. 4, pp. 1261–1282, apr 2005. [13] F . Reif, Fundamentals of Statistical and Thermal Physics . Mcgra w-Hill, 1965. [14] K. Huang, Statistical Mech anics . NY : John Wiley , 1967. [15] R. E . Blahut, Principles and Practice of Information Theory . Reading, MA: Addison-W esley , 1987. DRAFT