Average Entropy Functions

A v erage Entropy Functions Qi Chen, Chen He, Lingge Jiang, Qingc huan W ang Dept. of Electronic Eng . Shanghai Jiao T ong Univ . Shanghai, China 20 0240 Email: { cq0 94, chenhe, lgjiang, r6144 } @sjtu.edu .cn Abstract — THIS P APER IS ELIGIBLE FOR THE STUDENT P APER A W ARD . The closure of the set of entropy functions associated with n di screte variables, Γ ∗ n , is a con vex cone in (2 n − 1) -dimensional space, but its fu ll characterization remains an open p roblem. In th is paper , we map Γ ∗ n to an n -dimensional region Φ ∗ n by a v eraging the joint entropies with the same number of va riables, and show that the simpler Φ ∗ n can be characterized solely by the Shann on-type information i nequalities. I . I N T R O D U C T I O N Giv en an n -dimension al discrete random vector X = ( X 1 , . . . , X n ) , for each non-em pty subset α of N = { 1 , 2 , . . . , n } there is a joint entropy H ( X α ) with X α = ( X i ) i ∈ α , and the 2 n − 1 joint entr opies form the entropy function ( H ( X α )) α ⊆N ,α 6 = ∅ of X . W e can then defin e Γ ∗ n ⊆ R 2 n − 1 as the set o f all possible entr opy func tions inv olving n discrete rand om variables, and Γ ∗ n as its closure. A vector H ∈ R 2 n − 1 is called en tropic if H ∈ Γ ∗ n , and alm ost entropic if H ∈ Γ ∗ n [1]. All H = ( H α ) α ⊂N ,α 6 = ∅ ∈ Γ ∗ n satisfy th e following Shannon -type informatio n inequalities for any subsets α , β of N ( we let H ∅ = 0 fo r con venience): H α ≥ 0 , (1) H α ≤ H β , α ⊆ β , (2) H α + H β ≥ H ( α ∪ β ) + H ( α ∩ β ) . (3) Howe ver , (1)–(3) are n ot sufficient condition s for an H ∈ R 2 n − 1 to be almost entropic when n ≥ 4 [ 2]. In other words, denoting by Γ n the set of vectors in R 2 n − 1 satisfying (1)–(3), we have Γ ∗ n ( Γ n , n ≥ 4 . (4) A nu mber o f non- Shannon -type infor mation inequa lities sat- isfied by the memb ers o f Γ ∗ n have sub sequently been found in [2]–[4 ], but the full characterization of Γ ∗ n remains an open problem . In this paper , we will show that an a veraged version of Γ ∗ n can be more easily charac terized. Definition 1: For a vector H = ( H α ) α ⊂N ,α 6 = ∅ ∈ R 2 n − 1 , we define its av erage as Ψ( H ) , ( h 1 , . . . , h n ) , (5) where h k =  n k  − 1 P | α | = k H α . If H is the entr opy fu nction of rand om vector X , w e call h = Ψ( H ) the av erage entropy function . Ψ then m aps Γ ∗ n to the set Φ ∗ n , Ψ(Γ ∗ n ) of all av erage en tropy functions, Γ ∗ n to the closure Φ ∗ n , and Γ n to Φ n , Ψ(Γ n ) . From the definition (1)–(3) of Γ n , Φ n can be given by Φ n = { ( h 1 , . . . , h n ) | h k − 1 − 2 h k + h k +1 ≤ 0 , k = 1 , . . . , n } , (6) where we let h 0 = 0 an d h n +1 = h n for con venience. Φ ∗ n is obviously a subset of Φ n since Γ ∗ n ⊆ Γ n , but we will sho w that they ar e actually equal. In other words, Φ ∗ n is characterizab le solely with the Shannon -type information inequalities. Theor em 1: Φ ∗ n = Φ n . This theorem will be proved in the next sectio n. I I . P RO O F O F T H E T H E O R E M It is on ly necessary to prove that Φ n ⊆ Φ ∗ n . W e first introdu ce a on e-to-one transform to give Φ n a simpler form. Definition 2: For a vector h = ( h 1 , . . . , h n ) ∈ R n , we define its second-o rder d ifference as Θ( h ) = ( g 1 , . . . , g n ) , (7) where g k = h k − 1 − 2 h k + h k +1 , k = 1 , . . . , n , with h 0 = 0 and h n +1 = h n . Θ maps Φ ∗ n to Λ ∗ n , Θ(Φ ∗ n ) , Φ ∗ n to Λ ∗ n , and Φ n to Λ n , Θ(Φ n ) . From (6), we have Λ n = { ( g 1 , . . . , g n ) | g k ≤ 0 , k = 1 , . . . , n } . (8) As Ψ and Θ ar e both linear maps, an d Γ ∗ n is a convex cone [5], Φ ∗ n and Λ ∗ n are both con vex con es as well. The refore, to prove th at Φ n ⊆ Φ ∗ n or equ i valently Λ n ⊆ Λ ∗ n , it is suf ficient to prove that g k , (0 , . . . , 0 | {z } k − 1 , − a, 0 , . . . , 0) ∈ Λ ∗ n (9) for k = 1 , . . . , n an d som e a > 0 . In othe r words, f or each k we nee d to find a random vector X whose a verage en tropy function is h k , Θ − 1 ( g k ) = a · (1 , 2 , . . . , k , . . . , k ) . (10) This X can b e con structed from a Reed- Solomon code. Specifically , let q be a power -of-two larger than n , C be the codeword set of an ( n, k ) Reed-Solo mon cod e on GF( q ) , and X = ( X 1 , . . . , X n ) b e a r andom codeword u niformly distributed over C , then the entropy fu nction of X is (1 0) with a = lo g q , as sho wn below . Let j 1 , . . . , j n be distinct ind ices in 1 , . . . , n . Accord - ing to the prop erties of Reed-Solomon codes, gi ven any x ∗ j 1 , . . . , x ∗ j k ∈ GF( q ) , ther e exists a uniqu e x = ( x 1 , . . . , x n ) ∈ C with x j l = x ∗ j l , l = 1 , . . . , k . For any x ∗ j 1 ∈ GF( q ) , there are thus q k − 1 codewords x ∈ C with x j 1 = x ∗ j 1 , one for each value com bination on k − 1 other p ositions, an d since X is equal to each c odew ord with prob ability q − k , we have p ( X j 1 = x ∗ j 1 ) = q − 1 , so H ( X j 1 ) = log q . Similarly , H ( X j 1 , X j 2 ) = 2 log q , . . . , H ( X j 1 , . . . , X j k ) = k log q . For l = k + 1 , . . . , n , given x j 1 , . . . , x j l , there is eith er one match- ing codeword in C or none, ther efore p ( X j 1 = x j 1 , . . . , X j l = x j l ) is q − k on its sup port, and H ( X j 1 , . . . , X j l ) = k lo g q . Consequently , the a verage entro py fun ction of X is ( 10) with a = log q as desired, and for each l , all  n l  l -variable joint entropies of X that ar e being averaged actu ally hav e th e same value. I I I . D I S C U S S I O N Determination of Γ ∗ n is important due to its close conn ection to the capacity region of gene ral multi-source multi-sin k wired networks [6], [ 7], but this seems to be a difficult prob lem, and even if a full characterization is fo und, co mputation al difficulties d ue to Γ ∗ n ’ s h igh dimensionality and c omplex structure mig ht redu ce its usefu lness in p ractice [8]. What we have shown is th at the region Φ ∗ n obtained by av eraging th e k -variable joint en tropies has a m uch simpler structure: it is not affected by the n on-Shan non in formation in equalities, and the linear Reed-Solomon c odes used in the proof sugg est that the suboptimality of linea r network coding is also hidd en by this averaging. On one han d, this mean s that fur ther work on the char acterization of Γ ∗ n must focus on the variation among the k -variable entrop ies, not just their averages. On the other ha nd, many p ractically interesting network s h av e a somewhat symmetric stru cture, possibly in a statistical sense, and an appro priately a veraged version of Γ ∗ n (not nec essarily as simplistic as Φ ∗ n ) might provide a tractable method for the determinatio n of their capacity regions. A verage entropy function s are also clo sely related to the MAP EXIT functio ns d iscussed in e.g. [9] for large n . A C K N OW L E D G M E N T This paper was supp orted by National Natu ral Science Foundation of China Gran ts No. 6 07721 00 and No. 608 72017 . R E F E R E N C E S [1] R. W . Y eung, “ A frame work for linear informatio n inequa litie s, ” IEEE T r ans. Inf. Theory , vol . 43, pp. 1924–1934, Nov . 1997. [2] Z. Zhang and R. W . Y eung, “On chara cteriza tion of entrop y function via informati on inequalit ies, ” IEE E T rans. Inf. Theory , vol. 44, pp. 1440– 1452, Nov . 1998. [3] X. Y an, R. Y eung and Z. Zhang, “ A class of non-Shannon type infor - mation inequalit ies an d their applicat ions, ” IEEE Int. Symp. Inf. Theory , W ashington, DC, June 2001. [4] R. Doughter , C. Freiling and K. Zeger , “Six new non-Shannon information inequal ities, ” IEEE Int. Symp. Inf. Theory , Seattle, W A, June 2006. [5] Z. Zhang and R. W . Y eung, “ A non-Shannon type conditional inequalit y of information quan tities, ” IEE E T ra ns. Inf . Theory , vol. 43, pp. 1982– 1986, Nov . 1997. [6] X. Y an, R. Y eung and Z. Zhang, “The capacity region for multi-source multi-sink network coding, ” IEEE Int. Symp. Inf. Theory , Nice, France, June 2007. [7] T . Chan and A. Grant, “Dualities between entropy functions and network codes, ” IEEE T rans. Inf. Theory , vol. 54, no. 10, pp. 4470–4487, Oct. 2008. [8] F . Mat ´ uˇ s, “Infinitely many information in equalit ies, ” IEEE Int. Symp. Inf. Theory , Nice, France, June 2007. [9] C. Measson, A. Monta nari, and R. Urbanke, “Maxwe ll construc tion: The hidden bridge between iterati ve and m aximum a posteri ori decoding, ” Jun. 2005, arXiv:c s.IT/0506083.