Joint source-channel coding via statistical mechanics: thermal equilibrium between the source and the channel

arXiv:0810.2164v1 [cs.IT] 13 Oct 2008 Joint Source–Channel Coding via Statistical Mechanics: Thermal Equilibrium Between the Source and the Channel ∗ Neri Merhav Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, ISRAEL Abstract We examine the classical joint source–channel coding problem from the viewpoint of sta- tistical physics and demonstrate that in the random coding regime, the posterior probability distribution of the source given the channel output is dominated by source sequences, which exhibit a behavior that is highly parallel to that of thermal equilibrium between two systems of particles that exchange energy, where one system corresponds to the source and the other corresponds to the channel. The thermodynamical entopies of the dual physical problem are analogous to conditional and unconditional Shannon entropies of the source, and so, their bal- ance in thermal equilibrium yields a simple formula for the mutual information between the source and the channel output, that is induced by the typical code in an ensemble of joint source–channel codes under certain conditions. We also demonstrate how our results can be used in applications, like the wiretap channel, and how can it be extended to multiuser scenar- ios, like that of the multiple access channel. Index Terms: joint source–channel coding, statistical physics, thermal equilibrium, mutual information, entropy. 1 Introduction Consider the following two seemingly unrelated problems, which serve as simple special cases of a more general setting we study later in this paper: The first is an elementrary problem in statistical physics: We have two subsystems of particles which are brought into thermal equilibirium with each other as well as with the environment (a ∗Part of this work was carried out during a visit in Hewlett–Packard Laboratories, Palo Alto, CA, U.S.A., in the Summer of 2008. 1 heat bath) at temperature T. The first subsystem consists of N particles having magnetic moments (spins), {si}, each of which may be oriented either in the direction of an applied external magnetic field B, in which case si = +1, or in the opposite direction, in which case si = −1, and its energy in both cases is given by −siB (up to a certain multiplicative constant, which carries the appropriate physical units, and which is irrelevant for the purpose of this discussion). In the second subsystem, there are n non–interacting particles {s′ i}n i=1, each one of which may lie in one of two possible states: the state s′ i = 0, in which the particle has zero energy, and the state s′ i = 1, in which it has energy e0. What is the average energy possessed by each one of these subsystems in equilibrium, as functions of e0, T, n, N, and B? The second problem is in Information Theory, in particular, it is in joint source–channel coding, where some of the notation used is deliberately chosen to be the same as in the previous paragraph: A binary memoryless source generates a vector s of symbols (s1, s2, . . . , sN), si ∈{+1, −1}, i = 1, . . . , N, with probabilities q = Pr{Si = +1} and 1 −q = Pr{Si = −1}. This vector is encoded into a binary channel codeword x(s) of length n and transmitted over a binary symmetric channel (BSC) with a crossover probability p < 1/2, and a binary n–vector y is received at the channel output. Consider the posterior distribution P(s|y) = P(s)W(y|x(s)) P s′ P(s′)W(y|x(s′)) where P(s) and W(y|x) are the probability distributions that govern the source and the chan- nel, respectively, as described above. Thus, clearly, P(s|y) is proportional to P(s)W(y|x(s)), or equivalently, ln P(s|y) is (within a term that is independent of s) given by ln P(s) + ln W(y|x(s)). For a typical code drawn uniformly at random from the ensemble of codes, what are the relative contributions of the source and the channel to this quantity, for those vectors s that dominate P(s|y) (i.e., those that capture the vast majority of the posterior probability)? It turns out, as we shall see in Section 3 below, that the two problems have virtually identical answers (in a sense that will be made clear and precise therein), provided that the parameters T and B of the first problem are related to the parameters p and q of the second problem by p = exp{−e0/kT} 1 + exp{−e0/kT} (1) 2 and q = exp{B/kT} 2 cosh(B/kT), (2) or equivalently, e0 = kT ln 1 −p p (3) and B = kT 2 ln q 1 −q, (4) where k is Boltzmann’s constant. Thermal equilibrium between the two subsystems in the above described physical problem, dic- tates a certain balance between their thermodynamical entropies in order to arrive at the maximum total entropy (by the second law of thermodynamics) for the total energy possessed by the entire system at the given temperature T. As the thermodynamical entropy, in its statistical–mechanical definition, is intimately related to the Shannon entropy, it turns out that this equilibrium relation between the thermodynamical entropies of the physic

…(Full text truncated)…

This content is AI-processed based on ArXiv data.