A Projection Method for Derivation of Non-Shannon-Type Information Inequalities

A Projection Met hod for Deri v ation of Non-Shannon -T ype I nformation Inequaliti es W eidong Xu Shanghai K ey Lab . Digital Media Proc. an d Trans. Dept. of Electrical E ng. Shanghai Jiao T ong Univ . Shanghai, 2 00240 , C hina Email: weidongxu@sjtu.ed u.cn Jia W ang Shanghai Key Lab . Digital Media Proc. and T r ans. Dept. of Electrical E ng. Shanghai Jiao T ong Univ . Shanghai, 2 00240 , China Email: jiawang@sjtu.edu.cn Jun Sun Shanghai Key Lab . Digital Media Proc. an d Trans. Dept. of Electrical Eng . Shanghai Jiao T ong Univ . Shanghai, 2 00240 , China Email: sunjun@cdtv .o rg.cn Abstract —In 1998, Zhang and Y eung f ound the first un- conditional non-Shann on-type inf ormation i nequality . Recently , Dougherty , Freiling and Zeger gav e six new u nconditional non- Shannon-type inf ormatio n inequalities. This work generalizes their work and pro vides a method to systematically derive non- Shannon-type informa tion in equalities. An application of this method rev eals new 4-va riable non-S hannon-type inf ormation inequalities. I . I N T RO D U C T I O N Let n be a positi ve integer , N the set { 1 , 2 , · · · , n } , and P ( N ) its power set. Let ξ = ( ξ i ) i ∈ N be an n - dimension discrete random vector . Let ξ I = ( ξ i ) i ∈ I , ∀ I ⊆ N . For conv enience, let ξ ∅ be a r andom variable takin g a fixed value with p robability 1. The entr opy function H ξ of ξ = ( ξ i ) i ∈ N maps sets I ⊆ N to the Shan non entr opies H ( ξ I ) , which take value on [0 , + ∞ ] . An en tropy function which takes finite values can be co nsidered as a vector in the Eu clidean space R P ( N ) . Let H ent N be the set o f a ll such entr opy fun ctions. Therefo re H ent N can be vie wed as a r egion in R P ( N ) . It is known that the closure of H ent N , cl ( H ent N ) , is a c onv ex cone [1]. A real function f o n R P ( N ) is an info rmation ineq uality if an d on ly if f ( x ) ≥ 0 for all x ∈ H ent N . By Shanno n-type info rmation inequalities, entropy function has the following pr operties: H ξ is normalized , H ξ ( ∅ ) = 0 ; nonde creasing, H ξ ( I ) ≤ H ξ ( J ) for I ⊆ J ⊆ N ; submo dular, H ξ ( I ) + H ξ ( J ) ≥ H ξ ( I ∪ J ) + H ξ ( I ∩ J ) for I , J ⊆ N . Let H N be the set o f vectors in space R P ( N ) satisfying the above three pr operties. Clearly , H N is a polyhedr al cone. Obviously , H ent N ⊆ H N . It is known that H ent 2 = H 2 , cl ( H ent 3 ) = H 3 , and cl ( H ent N ) 6 = H N for n ≥ 4 [1], [2 ]. Therefo re, H N is an o uter bo und of cl ( H ent N ) . In this work, all outer boun d H outer N referred will be ta ken for polyhedr al cone. Since Shanno n-typ e inform ation inequalities can not fully characterize cl ( H ent N ) , there must e xist non-Sh annon -type informa tion ineq ualities. First unconditional lin ear non- Shannon -type information inequality was fo und in [2 ]. More of th ese non-Shanno n-type inf ormation ineq ualities ap peared in [3 ]–[6] . V ery rec ently , Ma t ´ u ˇ s [17 ] derived an infinite se- quence of new 4-variable and 5-variable linear in formatio n inequalities, and used them to show that cl ( H ent N ) is not a polyhed ral cone, for a ll n ≥ 4 . That is to say , cl ( H ent N ) can n ot be fully characterized b y a finite n umber of linear inf ormation inequalities. In the process of deriving n on-Sha nnon- type info rmation inequalities in [2] and [5], the auth ors used similar method s. This work generalize s their concept and method , th en brings forward a method to systematically derive n on-Shan non- type informa tion inequalities. In p articular, using this method, we find new 4 -variable un conditio nal lin ear non-Sh annon- type informa tion inequalities. I I . P O LYH E D R A L C O N E A N D I T S P R O J E C T I O N In this section, we first briefly re view the definition an d proper ties of poly hedral c one, then turn to the projection o f polyhed ral c one. A po lyhedr al cone P in Euclidean spac e R n can be rep re- sented in two ways: by the in tersection o f a finite number of closed half -spaces (an H -represen tation), P = { x ∈ R n | Ax ≥ 0 } , where A ∈ R r × n ; or by the nonnegative linear combina tion of a finite set of extreme rays (a V -represen tation), P = { x ∈ R n | x = Ry , y ∈ R s , y ≥ 0 } , where R ∈ R n × s . Minkowski-W eyl Th eorem states that these two representa - tions are equiv alent [10]. In th is work , a polyhedral con e is vie wed as a region in Euclidean space as well as the finite set of linear inequ alities representin g th e closed half-spa ces definin g the region. In this view , polyhedral cone H N can be consider ed as the region in space R P ( N ) as well a s a set of facet-defining Shannon - type in formatio n in equalities, n amely , elemental for ms of Shannon ’ s inform ation measures [8 ]. A projection of a region P in R n = R n 1 × R n 2 onto its subspace R n 1 is π x 1 ( P ) =  x 1 ∈ R n 1     ∃ x 2 such that x =  x 1 x 2  ∈ P  . That is to say , π x 1 is a pro jection dete rmined b y its rang e R n 1 and null space R n 2 . And π x 1 ( P ) is the set o f all the vectors π x 1 ( x ) , x ∈ P . The projectio n of a polyh edral cone is still a p olyhed ral cone. I n its V -represen tation, projection of P onto R n 1 is simply π x 1 ( P ) = { x 1 ∈ R n 1 | x 1 = R 1 y , y ∈ R s , y ≥ 0 } , where R 1 ∈ R n 1 × s , R 2 ∈ R n 2 × s ,  R 1 R 2  = R . The projection of a poly hedral cone in its H -represen tation is m ore difficult. W e will d etail it later in Section IV. Now we discuss some results relating a polyhed ral cone with th e half- spaces and vertex es o f its projection. A homog eneou s linear inequality ax ≥ 0 can be inferred from a system of homogeneo us lin ear inequalities Ax ≥ 0 if and o nly if the result o f the linear pr ogramm ing pr oblem minimize ax subject to Ax ≥ 0 is g reater than or eq ual to 0 , if and on ly if there exists a vector y ≥ 0 so that y T A = a . The latter equ iv alence is in fact th e well-known Farkas Lem ma. Line ar p rogram ming can b e u sed either to verify the former linear optimization problem or to find a nonnegative so lution o f the latter linear equations. A set of h omoge neous linear inequ alities is c alled in depen- dent, if none of its inequalities can b e infer red from oth er inequalities in the set. Lemma 1. A linear inequ ality ax 1 ≥ 0 can b e in ferr ed fr om the pr ojec tion of a set of linear inequa lities A 1 x 1 + A 2 x 2 ≥ 0 onto R n 1 if and o nly if it can be dir ectly in ferr ed fr o m A 1 x 1 + A 2 x 2 ≥ 0 . The lemma results from Projection Lemma used in block elimination algorith m [12] , which is proved u sing Farkas Lemma. Lemma 2. A vertex x 1 is in the pr ojection of a set of linear inequalities A 1 x 1 + A 2 x 2 ≥ B if and o nly if the r esult o f the linear pr ogramming pr oble m maximize ( B − A 1 x 1 ) T y subject to A T 2 y = 0 , y ≥ 0 is less than o r equ al to 0 . The equ iv alence can be r eadily p roved by Farkas Lemma. Owing to th e ab ove two lem mas, with the knowledge of the extreme ray s and linear in equalities of a polyhed ral c one P 1 , one can verify whethe r it is the projectio n of polyhe dral cone P , withou t actually com puting the projection. I I I . T H E O R E T I C A L B A C K G RO U N D In th is sectio n, w e present th e theo retical b ackgro und of the work. In 2 n -dimension Eu clidean spac e R P ( N ) , a vector is written as x P ( N ) = ( x I ) I ∈P ( N ) . In the discussion o f linear informa- tion ineq uality , x ∅ is fixed to be 0 . A. Pr ojectio n of H N and H ent N Definition 1. A pr o jection of a re gion P in R P ( N ) onto its sub- space R P ( M ) is π x P ( M ) ( P ) , where M = { 1 , 2 , · · · , m } , m < n . This is the same as the concept of restriction in [6], [17] . Proposition 1. π x P ( M ) ( H ent N ) = H ent M . π x P ( M ) ( cl ( H ent N )) = cl ( H ent M ) . The proposition can be easily p roved by truncatin g an n - dimension discr ete rand om vector . Proposition 2 . π x P ( M ) ( H N ) = H M . B. Pr oce dur e for Derivation o f Non-Sha nnon -T ype Info rma- tion In equalities Follo wing, we state th e proc edure for systematically deri v- ing no n-Shann on-typ e infor mation inequalities. Below , we re- strict consideratio n t o rand om vectors whose entropy functions take finite values. Say , we want to find m -variable infor mation inequalities. For a ny m -dimension rando m vecto r ξ , we constru ct a corre- sponding n -dim ension ( n > m ) random vector ζ = ( ζ i ) i ∈ N , so that th e marginal distribution of its first m r andom v ariables, namely , the p robab ility distrib ution of ζ M , equals that of ξ . Let ∆ N in R P ( N ) be the set of entro py functio ns of all these n -dimensio n d iscrete random vectors. Observing the above construction an d recalling the definition of projection, it fol- lows that π x P ( M ) (∆ N ) = H ent M . Th erefor e, if we can to some extent character ize ∆ N , thu s its projection π x P ( M ) (∆ N ) , we can finally arrive at an ou ter bo und of H ent M . Obvio usly , H N is a tri v ial ou ter bound of ∆ N . Since π x P ( M ) ( H N ) = H M , it p rovides no new inform ation about H ent M . Howe ver, for proper ly designed ζ , som e better outer bo und of ∆ N can be reached. In the latter part of the section, we f ocus on th e tech nique used in [5] to construc t the n -d imension rand om vector ζ a nd the oute r bou nd of ∆ N . W e restate a lemma in [9, L emma 14.8 ]. Lemma 3. Given an n -d imension random vector ξ = ( ξ i ) i ∈ N , ther e exists a r andom varia ble ξ n +1 , such that: 1) The join t pr obability distribution of ( ξ n +1 , ξ I ) equ als that of ( ξ k , ξ I ) , 2) Markov chain ξ { k }∪ J → ξ I → ξ n +1 holds, wher e { k } , I , J ar e disjoint sub sets of N . In [5] , random variable ξ n +1 is called a ξ J -copy of ξ k over ξ I . Therefo re, with ξ n +1 , ζ = ( ξ 1 , · · · , ξ n , ξ n +1 ) is an n + 1 - dimension ran dom vector . And H ζ ( { n + 1 } ∪ I 1 ) = H ζ ( { k } ∪ I 1 ) , I 1 ⊆ I , H ζ ( { n + 1 } ∪ I ) + H ζ ( I ∪ J 1 ) = H ζ ( { n + 1 } ∪ I ∪ J 1 ) + H ζ ( I ) , J 1 ⊆ { k } ∪ J, J 1 6 = ∅ , for all ζ . For constru cting an n -dimen sion random vector from m - dimension ran dom vector ξ , we start with ξ , th en add aux iliary random variables ξ m +1 , · · · , ξ n accordin g to Lemma 3, choos- ing parameters k , I , J e very time. The pro cess end s with an n -dimensio n rand om vector ζ = ( ξ 1 , · · · , ξ m , ξ m +1 , · · · , ξ n ) . Besides all n -variable Shannon -type informa tion ineq ualities, depend ing on the parameter s chosen, the entropy fu nction of ζ also satisfies the above two kinds of equalities. Le t C N denote th e set of a ll these ad ditional linear equalities. The region of the entropy functions o f all ζ , ∆ N , is outer bound ed by the polyh edral con e H N ∩ C N . Recalling that H ent M = π x P ( M ) (∆ N ) , it fo llows that H ent M ⊆ cl ( H ent M ) ⊆ π x P ( M ) ( H N ∩ C N ) . The pr ojection o f H N ∩ C N can b e explicitly calculated. If π x P ( M ) ( H N ∩ C N ) is strictly smaller than H M , we then hav e a b etter characterization of H ent M . In othe r words, some of the facet-definin g linear inequalities of π x P ( M ) ( H N ∩ C N ) must be m -variable n on-Shan non- type informa tion inequalities. In the above proc edure, alternatively , we co uld use any known ou ter b ound H outer N instead of H N . That is to say , we calculate the projection of H outer N ∩ C N instead. This means using kn own linear non-Shan non- type inform ation inequali- ties in the deri vation of new no n-Shan non-ty pe in formatio n inequalities. W e delay the application of above pro cedure to Section V. I V . P RO J E C T I O N A L G O R I T H M Although projection of a polyhed ral cone in its V - representatio n is simply the projectio n of all its extreme r ays, in this work, we p refer the H -represen tation, par tly because we are interested in linear info rmation inequa lities, and partly because the V -represen tation of an o uter bo und migh t be exponentially more co mplex than its H -represen tation. In o rder to deri ve non-Shan non-ty pe info rmation inequal- ities using th e theo ry and method men tioned in Section III, we must actually calculate th e projection of a poly hedral cone. Howe ver , e ven for deriving 4 -variable non- Shanno n- type information inequalities by projecting the po lyhedra l cone outer bou nding the entropy fun ctions of all correspond ing 6 - dimension rando m vecto rs, it requires calculating the pro jec- tion o f a 2 6 -dimension po lyhedr al cone consisting of hu ndred s of linear inequalities o nto a 2 4 -dimension space. Moreover , there is no e xplicit complexity results known for polyhedr on projection [ 12]. Classical projection m ethods such as Fourier-Motzkin elim- ination and block elimination are too com putationa lly in effi- cient to project hig h dimen sion polyhed ron [11], [12 ]. And the degeneracy of the polyhe dral cone may render other pro jection algorithm s, such as ESP [1 5], inefficient. W e turn to con vex h ull m ethod (CHM) propo sed in [13]. It works directly in th e projectio n spac e. The flowchart of CHM is depicte d in Fig. 1 . For p rojecting a p olyhedr al cone P in R n onto R n 1 , the algo rithm incrementally construc ts a polyhedr al cone P 1 in R n 1 while maintain ing its d ouble description p air  A 1 A 2  , R 1  . Th e ou tput is π x 1 ( P ) = P 1 , rep resented by Construct an initial P 1 with facets A 1 and extreme rays R 1 Input: Facets A of P Is A 1 empty? Choose a facet a from A 1 no Is a a facet of π ( P )? Move a outward to find extreme ray r Add r to R 1 Update P 1 , A 1 by R 1 using convex hull agorithm no yes Output: Facets A 2 and extreme rays R 1 of π ( P ) Move a from A 1 to A 2 yes Fig. 1. Flowc hart of conv e x hull method for projection double de scription pair ( A 2 , R 1 ) . The conve x h ull algorithm used in the CHM can be imple- mented using any incre mental algorithm . W e adopt Fourier- Motzkin elimination (the dual o f double descriptio n method [14]), since it appea rs to deal well w ith d egeneracy [16 ]. W e can take advantage of the increm ental nature of CHM to f urther reduce its compu tational complexity . Let H inner M be an inner bo und of cl ( H ent M ) . Then th e extreme rays of H M in H inner M must be the extreme rays o f cl ( H ent M ) , as well as the extreme rays of any π x P ( M ) ( H outer N ∩ C N ) . If the p olyhed ral cone gener ated b y these extreme rays is full dim ensional (except on ax is x ∅ ), we can then skip the in itialization and immediately start CHM fr om this approxim ation. This m ethod can be used for p rojection onto R P ( { 1 , 2 , 3 , 4 } ) , since the inner bound of cl ( H ent 4 ) is known [2], [7]. V . N E W 4 - V A R I A B L E N O N - S H A N N O N - T Y P E I N F O R M A T I O N I N E Q UA L I T I E S In th is section, we focu s on 4 -variable non-Sh annon -type informa tion in equalities. They are of theoretica l importance, meanwhile, low dimension p olyhed ral cones are more compu- tationally tractable. In the following, we discuss two different approa ches for c alculating n ew non- Shanno n-type information inequalities using the procedure introduced in Section III-B. Then the results are used to help derive an infinite sequence of 4 - variable n on-Shan non- type information inequ alities. A. Pr ojectio n by Adding Mor e Ra ndom V ariables First we add the 5th and 6 th random variables, as what i s shown in the p roof of [ 5, Theor em I II.1]. For any probability distribution of 4 - dimension random vector ξ , by Lemma 3, let ξ 5 be a ξ 4 -copy of ξ 3 over ξ { 1 , 2 } , ξ 6 be a ξ 2 -copy of ξ 3 over ξ { 1 , 4 , 5 } . Th is resu lts in a 6 - dimension ra ndom vector ζ = ( ξ 1 , · · · , ξ 6 ) . And th e co rrespon ding p olyhed ral cone C 6 has 18 equa lities. π x P ( { 1 , 2 , 3 , 4 } ) ( H 6 ∩ C 6 ) is a po lyhedr al con e defined by 35 linear information in equalities, which reveals Zhang- Y e ung inequality and the thir d an d fifth Dou gherty- Freiling-Zeger inequalities. W e u se more th an 6 random variables. For instan ce, let ξ 7 be a ξ 4 -copy of ξ 2 over ξ { 1 , 3 , 5 , 6 } . And ζ = ( ξ 1 , · · · , ξ 7 ) . The correspo nding po lyhedra l con e C 7 has 19 more equal- ities. The r esulting projectio n π x P ( { 1 , 2 , 3 , 4 } ) ( H 7 ∩ C 7 ) is a polyhed ral cone de fined by 56 lin ear info rmation ineq ualities, which re veals 1 3 in depend ent non-Shann on-typ e informatio n inequalities un known bef ore. Due to spac e limitation s we ju st list several of th em. − 56 H ( ξ { 1 } ) − 4 H ( ξ { 2 } ) − 19 H ( ξ { 3 } ) + 45 H ( ξ { 1 , 2 } ) + 67 H ( ξ { 1 , 3 } ) + 22 H ( ξ { 2 , 3 } ) + 23 H ( ξ { 1 , 4 } ) − 8 H ( ξ { 2 , 4 } ) + 9 H ( ξ { 3 , 4 } ) − 55 H ( ξ { 1 , 2 , 3 } ) − 24 H ( ξ { 1 , 3 , 4 } ) ≥ 0 , − 34 H ( ξ { 1 } ) − 2 H ( ξ { 2 } ) − 11 H ( ξ { 3 } ) − H ( ξ { 4 } ) + 27 H ( ξ { 1 , 2 } ) + 40 H ( ξ { 1 , 3 } ) + 12 H ( ξ { 2 , 3 } ) + 15 H ( ξ { 1 , 4 } ) − 5 H ( ξ { 2 , 4 } ) + 7 H ( ξ { 3 , 4 } ) − 32 H ( ξ { 1 , 2 , 3 } ) − 16 H ( ξ { 1 , 3 , 4 } ) ≥ 0 , − 28 H ( ξ { 1 } ) − H ( ξ { 2 } ) − 10 H ( ξ { 3 } ) − 2 H ( ξ { 4 } ) + 22 H ( ξ { 1 , 2 } ) + 34 H ( ξ { 1 , 3 } ) + 11 H ( ξ { 2 , 3 } ) + 13 H ( ξ { 1 , 4 } ) − 4 H ( ξ { 2 , 4 } ) + 6 H ( ξ { 3 , 4 } ) − 28 H ( ξ { 1 , 2 , 3 } ) − 13 H ( ξ { 1 , 3 , 4 } ) ≥ 0 . The novelty and independence of these information in equal- ities can be verified b y finding an extreme ray which does not satisfy one of th e ineq ualities, but satisfies all 4 -variable Shannon -type in formatio n in equalities together with all sub- stituted f orms of Zhan g-Y eung inequality , Doug herty-Freilin g- Zeger ineq ualities, Mat ´ u ˇ s inequalities, an d the re st of these inequalities. The verification can be d one by linear progr am- ming. Nev ertheless, with out calculation (pr ojection o r verifica- tion), we do n ot k now befor ehand how these auxiliary r andom variables would affect the resulting p rojection. Th is h inders us from systematically designing the random vector ζ . B. Pr ojectio n Using Impr oved O uter Boun d Alternatively , we choo se not to add that many aux iliary random variables, but to u se an outer bound strictly smaller than H N . Let ξ 5 be a ξ 4 -copy of ξ 3 over ξ { 1 , 2 } . Dur ing process in Section V -A , it is already known that π x P ( { 1 , 2 , 3 , 4 } ) ( H 5 ∩ C 5 ) only reveals Zhang-Y eung n on-Shan non-ty pe inform ation in- equality , − 2 H ( ξ { 1 } ) − 2 H ( ξ { 2 } ) − H ( ξ { 3 } ) + 3 H ( ξ { 1 , 2 } ) + 3 H ( ξ { 1 , 3 } ) + 3 H ( ξ { 2 , 3 } ) + H ( ξ { 1 , 4 } ) + H ( ξ { 2 , 4 } ) − H ( ξ { 3 , 4 } ) − 4 H ( ξ { 1 , 2 , 3 } ) − H ( ξ { 1 , 2 , 4 } ) ≥ 0 . (1) An outer bo und H outer (1) 4 is constru cted from H 4 and Zhang - Y e ung inequ ality . Throu gh substitution , a n outer bound H outer (1) 5 can be constructed f rom H outer (1) 4 . W e repeat the procedu re, but project H outer (1) 5 ∩ C 5 rather than H 5 ∩ C 5 . The r esulting π x P ( { 1 , 2 , 3 , 4 } ) ( H outer (1) 5 ∩ C 5 ) is a polyhedra l con e defined by 55 linear informatio n inequalities, 6 of wh ich are indepen- dent inf ormation ine qualities that can not be infer red fr om H outer (1) 4 . Am ong them, the following is un known be fore, − 10 H ( ξ { 1 } ) − 10 H ( ξ { 2 } ) − H ( ξ { 3 } ) + 17 H ( ξ { 1 , 2 } ) + 10 H ( ξ { 1 , 3 } ) + 10 H ( ξ { 2 , 3 } ) + 4 H ( ξ { 1 , 4 } ) + 4 H ( ξ { 2 , 4 } ) − 3 H ( ξ { 3 , 4 } ) − 16 H ( ξ { 1 , 2 , 3 } ) − 5 H ( ξ { 1 , 2 , 4 } ) ≥ 0 . (2) Outer bound H outer (2) 4 is co nstructed from H outer (1) 4 together with these new inf ormation inequ alities. It can be o bserved that the above proced ure is indeed a functio n wh ich maps H 4 to H outer (1) 4 , and subsequ ently H outer (1) 4 to H outer (2) 4 . C. Infinite Seq uence of 4 -variable Non -Shan non-T ype Infor - mation Ineq ualities Thoug h the projection alg orithm will on ly rev eal finite number of non-Shanno n-type linear information inequalities, the in formatio n acquired from the resu lts may still help shed some lig ht o n th e infin ite sequ ence of linear infor mation inequalities. Once a new m -variable linear infor mation in equality is derived by pr ojecting some H outer N ∩ C N , it ca n be written as a no nnegative line ar combination of those n -variable in- formation inequalities a nd additional equalities, as discussed in Section II. Th is nonn egati ve lin ear co mbinatio n can b e viewed as an exp licit proof o f this newly d erived inform ation inequality . It can be observed that Zhang-Y eung information ineq uality (1) and in formation inequality ( 2) b ear some sim ilarity . More- over , the explicit pro ofs correspon ding to th e two information inequalities shar e a similar structu re. Therefo re, with several more iterations and som e guesswork, we deriv e the following infinite seque nce of 4 -variable no n-Shann on-typ e inf ormation inequalities. 2 s − 1 − √ 2 4 S + + √ 2 4 S − !  H ( ξ { 1 } ) + H ( ξ { 2 } )  − H ( ξ { 3 } ) + 1 − 3 · 2 s − 1 + √ 2 2 S + − √ 2 2 S − ! H ( ξ { 1 , 2 } ) +  1 4 S + + 1 4 S −   H ( ξ { 1 , 3 } ) + H ( ξ { 2 , 3 } )  + √ 2 − 1 4 S + − √ 2 + 1 4 S − !  H ( ξ { 1 , 4 } ) + H ( ξ { 2 , 4 } )  +  1 − 2 s − 1  H ( ξ { 3 , 4 } ) +  2 s − 1 − 1 2 S + − 1 2 S −  H ( ξ { 1 , 2 , 3 } ) − 1 − 2 s − 1 + √ 2 − 1 2 S + − √ 2 + 1 2 S − ! H ( ξ { 1 , 2 , 4 } ) ≥ 0 , where S + =  2 + √ 2  s , S − =  2 − √ 2  s , and s ∈ N . (3) A rigoro us p roof ca n be easily obtained through mathematical induction . When s = 1 , information ine quality (3) correspo nds to a Shanno n-type info rmation ineq uality . For s = 2 and 3 , it is Zhang-Y eung ineq uality and information inequality (2), respectively . V I . D I S C U S S I O N A N D F U T U R E W O R K The concept and usage of projection is not limited to the theory and application men tioned above. Th e notion o f infe r- ence r ule in [3 ] an d in ner adhesivity in [17] an d th eir m ethods for de riving n on-Sha nnon- type infor mation inequ alities can also be regard ed as a practice of pr ojection. For example, in the pr oof of the 5-variable non -Shanno n-type information inequality in [3], the application of the inference rule is equiv alently the fo llowing set of add itional equalities, x { 3 , 4 , 5 } + x { 3 , 4 }∪ J − x { 3 , 4 , 5 }∪ J − x { 3 , 4 } = 0 , J ⊆ { 1 , 2 } , J 6 = ∅ , and the r equireme nt that no term in the resulting inequality has any of { 1 , 2 } togeth er with { 5 } specifies the range o f the pro jection, i.e. the subspa ce consisting o f all vectors with x I ∪ J ∪{ 5 } = 0 , I ⊆ { 3 , 4 } , J ⊆ { 1 , 2 } , J 6 = ∅ . The proced ure used in Section V -A can be generalized and its limit property is to be inv estigated. Let H outer M ↓ N denote the intersection of all projections π x P ( M ) ( H N ∩ C N ) derived using the pro cedure described in Sectio n III- B applying Lemma 3. For exam ple, it can be shown that H outer 4 ↓ 5 is in fact th e region enclosed b y all 4 -variable Shannon- type informatio n in equal- ities and all substituted forms of Zhang-Y eung inequ ality , and that H outer 4 ↓ 6 is the region enclosed by all 4 -variable Sh annon - type infor mation in equalities and all substituted for ms of Zhang- Y e ung ineq uality and Dough erty-Freiling -Zeger in- equalities. It is easy to see that H outer M ↓ N ⊆ H outer M ↓ N ′ , for n > n ′ . Th erefore, we d efine H outer M ↓ N = T n>m H outer M ↓ N . And obviously H outer M ↓ N ⊇ H ent M . Then, the critical issue is whether H outer M ↓ N = cl ( H ent M ) . If the above equation holds, then this work pr esents an explicit ch aracterization of cl ( H ent M ) , and we are capable of calculating and approx imating it. Oth erwise, the prob lem is whether ther e exist other metho ds of adding random variables and o ther kin ds o f ad ditional inequalities that can be used as C N , so that after a similar pro cess the newly obtained H outer M ↓ N would coinc ide with c l ( H ent M ) . Similarly , th e pr ocedure described in Section V -B can b e generalized . Let σ be a fun ction defined o n the set of p olyhe- dral con es in R P ( M ) , which m aps an ou ter bound H outer M to the intersection of all projections π x P ( M ) ( H outer N ∩ C N ) , fo r fixed n . Clearly , σ k +1 ( H M ) ⊆ σ k ( H M ) , and σ k ( H M ) ⊇ H ent M . Then, the prob lem is whether σ k +1 ( H M ) is strictly smaller than σ k ( H M ) , and whether lim k →∞ σ k ( H M ) = cl ( H ent M ) . These call for fu rther inv estigation. R E F E R E N C E S [1] Z. Z hang and R. W . Y eung, “ A non-Shannon-type conditiona l inequality of information quantities, ” IE EE Tr ans. Inform. Theory , vol. 43, no. 6, pp. 1982–1985, Nov . 1997. [2] Z. Zhang and R. W . Y eung, “On characte rizat ion of entrop y functi on via informati on ine qualit ies, ” IE EE T rans. Inform. Theory , vol. 44, no. 4, pp. 1440–1452, Jul. 1998. [3] K. Makaryche v , Y . Makaryche v , A. Romashchenk o, and N. V ereshcha- gin, “ A new class of non-Shannon-type ineq ualiti es for entropies, ” Commun. Inf. and Syst. , vol. 2, no. 2, pp. 147–166, Dec. 2002. [4] Z. Z hang, “On a ne w non-Shanno n type informati on inequality , ” Com- mun. Inf. and Syst. , vol. 3, no. 1, pp. 47–60, June 2003. [5] R. Dougherty , C. Frei ling, and K. Zeger , “Six ne w non-Shannon in- formation inequa liti es, ” in Proc. IE EE Int. Symp. Information Theory , Seattl e, W A, Jul. 2006, pp. 233–236. [6] F . Mat ´ uˇ s, “Inequali ties for Shannon entro pies and adhesi vity of polyma- troids, ” in Pr oc. 9th Canadi an W orkshop on Informat ion Theory , McGill Uni ver sity , Montr ´ eal, QC, Canada, 2005, pp. 28–31. [7] F . Mat ´ u ˇ s and M. Studen ´ y, “Condi tional indepen dences among fou r random va riable s I, ” Combinatorics, Prob abilit y and Computing , vol. 4, no. 4, pp. 269–278, 1995. [8] R. W . Y e ung, “ A framew ork for linear information inequa litie s, ” IE EE T rans. Inform. Theory , vol . 43, no. 6, pp. 1924–1934, Nov . 1997. [9] R. W . Y e ung, A Fi rst Cour se in Information Theory . New Y ork: Kluwer Academic /Plenum, 2002. [10] G. M. Ziegl er , Lectur es on P olytopes . Springer-V erlag, 1995. [11] T . Huynh, C. Lassez, and J-L. Lassez, “Prac tical issues on the projecti on of polyhedral sets, ” A nnals of Mathematics and Artificia l Intellig ence , vol. 6, pp. 295–315, Nov . 1992. [12] B. Kaluzny , “Polyhedr al computati on: a surve y of proje ction methods, ” Class Project, 2002. [13] C. Lassez and J-L. Lassez, “Quantifier elimina tion for conjunc tions of linea r constrai nts via a con ve x hull algorithm, ” Symbolic and Numerical Computati on for Artificial Intell igen ce , Donald, Kapur , and Mundy Ed., Academic Press, 1993. [14] K. Fukuda and A. Prodon, “Double description method re visited , ” Combinato rics and Comput er Science , vol. 1120 of LNCS. Springe r- V erl ag, pp. 91–111, 1996. [15] C. N. Jones, E. C. Kerriga n, and J. M. Maciejo wski, “Equalit y Set Project ion: A ne w algorit hm for the project ion of polytope s in halfspace represent ation, ” T echni cal Report CUED/F-INFENG/TR.463, 2004. [16] D. A vis, D. Bremner , and R. Seidel, “How good are con vex hull algorit hms?” Computational Geometry , vol. 7, no. 5, pp. 265–301, April 1997. [17] F . Mat ´ u ˇ s, “Infinite ly many information inequalitie s, ” in Pro c. IEEE Int. Symp. Information Theory , Nice, F rance, June 2007, pp. 41–44.