Hash Property and Fixed-rate Universal Coding Theorems

1 Hash Property and Fix ed-rate Uni v er sal Coding Theorems Jun Muramatsu Member , I EEE, Shigeki Miyake Member , IEEE, Abstract The aim of this paper is to prov e the achiev ability of fixed-rate univ ersal coding problems by using our pre viously introduced notion of h ash property . These problems are the fixed-rate lossless uni v ersal source coding problem and the fi xed-rate uni versal channel coding problem. S ince an ensemble of sparse matrices satisfies the hash property requirement, it is prov ed that we can construct uni versal codes by using sparse matrices. Index T erms channel coding, fixed-rate univ ersal codes hash functions, l inear codes, lossless source coding, mi nimum- di verg ence encoding, minimum-entropy decoding, shannon theory , sparse matrix I . I N T RO D U C T I O N The notion of hash property is intr oduced in [1 2]. I t is a sufficient condition fo r the a chiev ability of cod ing theorems including lossless and lossy sour ce cod ing, chann el codin g, the Slepian-W o lf pro blem, the W yner-Zi v problem , the Gel’fand-Pin sker problem, and the prob lem of sour ce coding with partial side informa tion at the decoder . Sin ce an ensem ble of sparse matrices satisfies the hash proper ty requiremen t, it is proved that we can construct codes by using sparse matrices and maximu m-likelihood codin g. Howe ver , it is a ssumed in [ 12] that sou rce an d channel distributions are used when desig ning a cod e. The aim of th is pap er is to prove fixed-rate universal co ding theorem s based on the hash pr operty , where a specific probab ility distribution is not assum ed for the design of a code and the error probab ility of a co de v anishes for all sources specified by the encod ing rate. W e prove theo rems o f fixed-rate lossless un iv ersal sou rce coding (see Fig. 1) an d fixed-rate universal channel coding (see Fig. 2). I n the constru ction o f codes, the maximu m-likelihood coding used in [12] is replaced by a minimum- diver gence encoder an d a minim um-entr opy decoder . A practical algo rithm has been o btained for the minimum- entropy decoder b y usin g linear pro grammin g [2]. It shou ld be noted that a practica l alg orithm f or the minimum -diver gence encoder can also b e obtained by using lin ear prog ramming as shown in Section V. The fixed-r ate lo ssless universal sourc e coding the orem is pr oved in [3] for the ensemble of a ll linea r matrices in the context of the Slepian-W olf source coding p roblem, in [7] for the class of universal hash fun ctions, and J. Muramatsu is with NTT Communicatio n Science Laborato ries, NTT Corporati on, 2-4, Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan (E-m ail: pure@csla b .kecl .ntt.co.jp). S. Miyake is with NTT Netwo rk Innov ation Laboratori es, NTT Corporation, 1-1, Hikarino oka, Y okosuka-shi, Kanaga wa 239-0847, Japan (E-mail: miyake.shige ki@lab .ntt.co.jp). Novem ber 14, 2018 DRAFT 2 X ✲ ϕ ✲ R > H ( X ) ϕ − 1 ✲ X Fig. 1. Lossless Source Coding M ✲ ϕ ✲ X ✲ µ Y | X ✲ Y ✲ ϕ − 1 ✲ M R < I ( X ; Y ) Fig. 2. Channel Coding in [11] implicitly for an ensemble of sparse matrices in the context of a secret key agreement from correlated source o utputs. The universal chann el co ding theor em that em ploys sparse matrice s is p roved in [8] fo r an additive noise channel and in [9] for an arbitrary ch annel. It should be noted here that the linearity for an ensemble member is not assumed in our proo f. Our pr oof assumes that en sembles of sparse matrices have a hash proper ty and so is simpler than previously rep orted proofs [11][8][9]. I I . D E FI N I T I O N S A N D N OTA T I O N S Throu ghout this paper, we use the following defin itions and notations. Column vector s and sequ ences are denoted in bold face. Le t A u denote a value taken by a f unction A : U n → U at u ∈ U n where U n is a d omain of the function. It should be n oted that A may be n on-linear . For a function A and a set of function s A , let Im A and Im A be defin ed as Im A ≡ { A u : u ∈ U n } Im A ≡ [ A ∈A Im A. The cardinality of a set U is den oted b y |U | and U − { u } is a set difference. W e d efine sets C A ( c ) and C AB ( c , m ) as C A ( c ) ≡ { u : A u = c } C AB ( c , m ) ≡ { u : A u = c , B u = m } . In the context o f linear codes, C A ( c ) is called a coset determined by c . Let p an d p ′ be p robability distributions an d let q an d q ′ be co nditional pro bability distributions. Then entropy H ( p ) , con ditional en tropy H ( q | p ) , divergence D ( p k p ′ ) , and condition al diver gence D ( q k q ′ | p ) are defined as H ( p ) ≡ X u p ( u ) log 1 p ( u ) H ( q | p ) ≡ X u,v q ( u | v ) p ( v ) log 1 q ( u | v ) D ( p k p ′ ) ≡ X u p ( u ) log p ( u ) p ′ ( u ) Novem ber 14, 2018 DRAFT 3 D ( q k q ′ | p ) ≡ X v p ( v ) X u q ( u | v ) log q ( u | v ) q ′ ( u | v ) , where we assume the base 2 of the loga rithm. Let µ U V be the join t prob ability distribution of random variables U and V . Let µ U and µ V be the respective marginal distributions and µ U | V be th e con ditional pr obability distribution. Then the en tropy H ( U ) , the condition al entro py H ( U | V ) , an d the mutual information I ( U ; V ) of ran dom v ariables are defined as H ( U ) ≡ H ( µ U ) H ( U | V ) ≡ H ( µ U | V | µ V ) I ( U ; V ) ≡ H ( µ U ) + H ( µ V ) − H ( µ U V ) . Let ν u and ν u | v be defined as ν u ( u ) ≡ |{ 1 ≤ i ≤ n : u i = u }| n ν u | v ( u | v ) ≡ ν uv ( u, v ) ν v ( v ) . W e call ν u a typ e 1 of u ∈ U n and ν u | v a con ditional type. Let U ≡ ν U be th e typ e of a sequenc e and U | V ≡ ν U | V be the conditional type of a sequen ce giv en a sequ ence of type U . T hen a set of typ ical sequences T U and a set of condition ally typical sequ ences T U | V ( v ) are define d as T U ≡ { u : ν u = ν U } T U | V ( v ) ≡  u : ν u | v = ν U | V  , respectively . The empir ical entro py , th e em pirical conditio nal entr opy , and em pirical mutual info rmation are defined as H ( u ) ≡ H ( ν u ) H ( u | v ) ≡ H ( ν u | v | ν v ) I ( u ; v ) ≡ H ( ν u ) + H ( ν v ) − H ( ν uv ) . In the construc tion of a universal source code, we use a minimum-e ntr o py deco der g A ( c ) ≡ arg min x ′ ∈C A ( c ) H ( x ′ ) It sho uld be n oted that the linear progr aming technique intro duced in [2] ca n b e applied to the m inimum-e ntropy decoder g A . In the constru ction o f a universal ch annel code, we u se a minimum-dive r gence encoder g AB ( c , m ) ≡ arg min x ′ ∈C AB ( c , m ) D ( ν x ′ k µ X ) and a minimum -entropy decoder g A ( c , y ) ≡ arg min x ′ ∈C A ( c ) H ( x ′ | y ) . 1 In [12], the type of a sequence is defined as a histogram { nν u ( u ) } u ∈U . Novem ber 14, 2018 DRAFT 4 It should be noted that we have g AB ( c , m ) = a rg max x ′ ∈C AB ( c , m ) [log µ X ( x ′ ) + nH ( ν x ′ )] = arg max U ′  nH ( U ′ ) + max x ′ ∈C AB ( c , m ) ∩T U ′ log µ X ( x ′ )  from Lemma 7. When functions A and B are linear, the linear progr aming techn ieque in troduced in [6] can b e applied to the m aximization max x ′ µ X ( x ′ ) because U ′ is fixed and the constraint cond ition x ′ ∈ C AB ( c , m ) ∩ T U ′ is repre sented by linea r f unctions. Finally , we define χ ( · ) as χ ( a = b ) ≡      1 , if a = b 0 , if a 6 = b χ ( a 6 = b ) ≡      1 , if a 6 = b 0 , if a = b. W e define a seq uence { λ U ( n ) } ∞ n =1 as λ U ( n ) ≡ |U | log [ n + 1] n . (1) It shou ld be no ted here th at the product set U × V is denoted by U V when it appea rs in the su bscript of this function and we omit argument n o f λ U when n is clear in the context. W e d efine | · | + as | θ | + ≡      θ, if θ > 0 , 0 , if θ ≤ 0 . (2) I I I . ( α , β ) - H A S H P RO P E RT Y In this section, we rev eiw the notion of the ( α , β ) -h ash p roperty introd uced in [12]. This is a sufficient condition for cod ing theorems, where the linearity of fu nctions is not assumed . By using this no tion, we prove a fixed-rate u niversal sou rce coding theorem and a fixed-rate u niversal sou rce coding theorem. Throu ghout the paper, A u denotes a value taken by a fu nction A at u ∈ U n where U n is the domain o f the function . It should again be noted here that A may be non-lin ear . W e define the ( α , β ) -hash prope rty in the following. Definition 1: Let A b e a set of fu nctions A : U n → U and we assume that Im A = Im A for all A ∈ A and lim n →∞ log | U | | Im A| n = 0 . (H1) Let p A be a pr obability distribution on A . W e call a pair ( A , p A ) an ensemble . Then, ( A , p A ) has an ( α , β ) - hash pr o perty if α ≡ { α ( n ) } ∞ n =1 and β ≡ { β ( n ) } ∞ n =1 satisfy lim n →∞ α ( n ) = 1 (H2) lim n →∞ β ( n ) = 0 (H3) Novem ber 14, 2018 DRAFT 5 and X u ∈T u ′ ∈T ′ p ( { A : A u = A u ′ } ) ≤ |T ∩ T ′ | + |T ||T ′ | α ( n ) | Im A| + min {|T | , |T ′ |} β ( n ) (H4) for any T , T ′ ⊂ U n . Throu ghout this paper, we omit argument n o f α and β when n is fixed. In the following, we presen t two examples of ensembles that have a hash prope rty . Example 1: I n this examp le, we c onsider a u niversal class of hash functions introdu ced in [ 5]. A set A of function s A : U n → U is called a universal class of hash functions if | { A : A u = A u ′ } | ≤ |A| |U | for any u 6 = u ′ . For examp le, the set of all functions o n U n and the set of all linear fu nctions A : U n → U l A are universal classes of hash functio ns (see [5]). It sho uld be noted th at ev ery example above satisfies Im A = U . When A is a universal class of hash fu nctions and p A is the unifor m proba bility on A , we h av e X u ∈T u ′ ∈T ′ p A ( { A : A u = A u ′ } ) ≤ |T ∩ T ′ | + |T ||T ′ | | Im A| . This implies that ( A , p A ) has a ( 1 , 0 ) -hash property , where α ( n ) ≡ 1 and β ( n ) ≡ 0 fo r every n . Example 2: In this e xample, we reve w the ensemble of q - ary sparse matrices introduced in [12]. In the follo wing, let U ≡ GF( q ) an d l A ≡ nR . W e generate an l × n matr ix A with the fo llowing proced ure: 1) Start fr om an all-zero ma trix. 2) For each i ∈ { 1 , . . . , n } , repe at th e f ollowing proced ure τ times: a) Choo se ( j, a ) ∈ { 1 , . . . , l A } × [GF( q ) − { 0 } ] unif ormly at ran dom. b) Add a to the ( j, i ) com ponen t of A . Let ( A , p A ) be an ensemble corresp onding to the above p rocedur e. Then Im A =             u ∈ U l : u has an even number of non-ze ro elements    , if q = 2 U l , if q > 2 for a ll A ∈ A and th ere is ( α A , β A ) such that ( A , p A ) has an ( α A , β A ) -hash p roperty (see [12, Th eorem 2] ). In the following, Let A (resp. B ) be a set of function s A : U n → U A (resp. B : U n → U B ). W e assume that an ensemble ( A , p A ) has an ( α A , β A ) -hash property an d an ensemble ( A × B , p A × p B ) also has a n ( α AB , β AB ) -hash prop erty . W e a lso assum e th at p C and p M is the uniform distribution on Im A and Im B , respectively , and random variables A , B , C , and M are mutually indepen dent, that is, p C ( c ) =      1 | Im A| , if c ∈ Im A 0 , if c ∈ U − Im A p M ( m ) =      1 | Im B| , if m ∈ Im B 0 , if m ∈ U − Im A Novem ber 14, 2018 DRAFT 6 Encoder x ✲ A ✲ A x Decoder A x ✲ g A ✲ x Fig. 3. Construct ion of Fixe d-rate Source Code p AB C M ( A, B , c , m ) = p A ( A ) p B ( B ) p C ( c ) p M ( m ) for any A , B , and c . W e use the following lem mas, which are sh own in [12]. Lemma 1 ([12, Lemma 9]): For any A and u ∈ U n , p C ( { c : A u = c } ) = X c p C ( c ) χ ( A u = c ) = 1 | Im A| and for any u ∈ U n , E AC [ χ ( A u = c )] = X A, c p AC ( A, c ) χ ( A u = c ) = 1 | Im A| . Lemma 2 ([12, Lemma 2]): If G ⊂ U n and u / ∈ G , th en p A ( { A : G ∩ C A ( A u ) 6 = ∅} ) ≤ |G | α A | Im A| + β A . Lemma 3 ([12, Lemma 5]): If T 6 = ∅ , then p AB C M ( { ( A, B , c , m ) : T ∩ C AB ( c , m ) = ∅} ) ≤ α AB − 1 + | Im A|| Im B | [ β AB + 1] |T | . When ( A , p A ) and ( B , p B ) are the en sembles of l A × n a nd l B × n linear matrices, respectively , we ha ve the following lemma. Lemma 4 ([12, Lemma 7]): The joint d istribution ( A × B , p AB ) h as an ( α AB , β AB ) -hash p roperty for th e ensemble of functio ns A ⊕ B : U n → U l A + l B defined as A ⊕ B ( u ) ≡ ( A u , B u ) , where α AB ( n ) = α A ( n ) α B ( n ) (3) β AB ( n ) = min { β A ( n ) , β B ( n ) } . (4) Novem ber 14, 2018 DRAFT 7 I V . F I X E D - R AT E L O S S L E S S U N I V E R S A L S O U R C E C O D I N G In this section, we consider the fixed-rate lossless universal so urce coding illustrated in Fig. 1. For a g i ven en coding rate R , l A is giv en b y l A ≡ nR log |X | . W e fix a fu nction A : X n → X l A which is a vailable to constru ct an en coder an d a decoder . W e d efine the e ncoder a nd the decoder (illustrated in Fig. 3) ϕ X : X n → X l A ϕ − 1 : X l A → X n as ϕ ( x ) ≡ A x ϕ − 1 ( c ) ≡ g A ( c ) , where g A ( c ) ≡ arg min x ′ ∈C A ( c ) H ( x ′ ) . The error prob ability Err or X ( A ) is giv en by Erro r X ( A ) ≡ µ X  x : ϕ − 1 ( ϕ ( x )) 6 = x  . W e ha ve th e f ollowing theore m. It sho uld b e n oted that the alphabet X m ay not be binary . Theor em 1: Assum e that an ensemb le ( A , p A ) h as an ( α A , β A ) -hash p roperty . For a fixed rate R , δ > 0 and a sufficiently large n , there is a function (matrix) A ∈ A such that Erro r X ( A ) ≤ ma x  α A |X | l A | Im A| , 1  2 − n [inf F X ( R ) − 2 λ X ] + β A (5) for any stationary m emoryless so urces X satisfying H ( X ) < R, (6) where F X ( R ) ≡ min U ′  D ( ν U ′ k µ X ) + | R − H ( U ′ ) | +  and the infimum is taken over all X satisfying (6) . Since inf X : H ( X ) >R F X ( R ) > 0 , then the error prob ability goes to zero as n → ∞ for all X satisfy ing (6). Novem ber 14, 2018 DRAFT 8 W e can prove the coding theorem fo r a chann el µ Y | X with additive no ise Z ≡ Y − X by letting A and C A ( 0 ) = { x : A x = 0 } be a par ity chec k matrix an d a set of codew ords (chann el in puts), respec ti vely . Then the encod ing rate of this chan nel co de is g iv en b y log |C A ( 0 ) | n ≥ lo g |X | − R and the error prob ability is given as Erro r Y | X ( A ) ≡ 1 |C A ( 0 ) | X x ∈C A ( 0 ) µ Y | X ( { y : g A ( A y ) 6 = y − x } | x ) . Since z = y − x A z = A y − A x = A y , then the decoding o f channel input x fr om a syn drome A y is equ i valent to the decoding of source o utput z from its codeword A z by using g A . W e hav e the f ollowing corollar y . Cor olla ry 2: Assume that an ensemb le ( A , p A ) of linear functio ns has an ( α A , β A ) -hash p roperty . For a fixed r ate R , δ > 0 and sufficiently large n , there is a (sparse) matrix A ∈ A such that Erro r Y | X ( A ) ≤ ma x  α A |X | l A | Im A| , 1  2 − n [inf F Z ( R ) − 2 λ X ] + β A for any stationary m emoryless c hannel with additive noize Z satisfying log |X | − R < I ( X ; Y ) = log |X | − H ( Z ) , (7) where the infimum is taken over all Z satisfyin g (7) an d the error prob ability goes to zero as n → ∞ fo r all X satisfying (7). Remark 1: I t should be noted here that the condition (H2) can be replaced by lim n →∞ log α A ( n ) n = 0 . (8) By using the expurgation techniq ue d escribed in [1], we obtain an en semble of sparce matrices that have an ( α A , 0 ) -hash pro perty , wh ere (H2 ) is replaced by (8 ). Th is implies that w e ca n omit the te rm β A from the upper bound of the error prob ability . Remark 2: Sin ce a class of un iv ersal hash fu nctions with a unifo rm distribution and an ensemble of all lin ear function s ha s a ( 1 , 0 ) -hash proper ty , we obtain the same results a s those rep orted in [7] and [3], r espectiv ely , where F X represents the error e xpon ent function. When ( A , p A ) is an ensemb le of sparse matrices and ( α A , β A ) is defined prop erly , we have the same r esult as tha t found in [8]. V . F I X E D - R A T E U N I V E R S A L C H A N N E L C O D I N G The code for the c hannel coding problem (illustrated in Fig. 2) is giv en in the following (illustrated in Fig. 4). The id ea for the constru ction is drawn f rom [10][12][9]. W e giv e th e explicit construction of the en coder by using minimum -diver gence en coding, wh ich is not described in [10][12][9]. Novem ber 14, 2018 DRAFT 9 Encoder c ✲ m ✲ g AB ✲ x Decoder c ✲ y ✲ g A ✲ x ✲ B ✲ m Fig. 4. Construct ion of Channel Code For a g i ven R A , R B > 0 , let A : X n → X l A B : X n → X l B satisfying R A = log | Im A | n R B = log | Im B | n , respectively . W e fix fun ctions A , B and a vector c n ∈ X l A av ailable to constract an en coder and a d ecoder . W e define th e e ncoder and the decoder ϕ : X l B → X n ϕ − 1 : Y n → X l B as ϕ ( m ) ≡ g AB ( c , m ) ϕ − 1 ( y ) ≡ B g A ( c , y ) , where g AB ( c , m ) ≡ arg min x ′ ∈C AB ( c , m ) D ( ν x ′ k µ X ) g A ( c , y ) ≡ arg min x ′ ∈C A ( c ) H ( x ′ | y ) . Novem ber 14, 2018 DRAFT 10 The error prob ability Err or Y | X ( A, B , c ) is given by Erro r Y | X ( A, B , c ) ≡ X m , y p M ( m ) µ Y | X ( y | ϕ ( m )) χ ( ϕ − 1 ( y ) 6 = m ) , where p M ( m ) ≡      1 | Im B | , if c ∈ Im B 0 if c / ∈ Im B . It sh ould b e n oted th at Im B represents a set o f all messages and R B represents the encoding rate of a channel. W e ha ve th e f ollowing theore m. Theor em 3: Assum e that an en semble ( A , p A ) (re sp. ( A×B , p AB ) ) has an ( α A , β A ) -hash (resp. ( α AB , β AB ) - hash) prope rty . For a fixed rate R A , R B > 0 , a given input distribution µ X satisfying H ( X ) > R A + R B , (9) δ > 0 , an d a sufficiently large n , ther e are fu nctions (matrices) A ∈ A , B ∈ B , and a vector c ∈ Im A such that Erro r Y | X ( A, B , c ) ≤ α AB − 1 + β AB + 1 κ + 2 κ h max { α A , 1 } 2 − n [inf F Y | X ( R A ) − 2 λ X Y ] + β A i (10) for all µ Y | X satisfying H ( X | Y ) < R A , (11 ) where F Y | X ( R ) ≡ min V | U [ D ( ν V | U k µ Y | X | ν U ) + | R − H ( U | V ) | + ] , the infimum is taken over all µ Y | X satisfying (9), and κ ≡ { κ ( n ) } ∞ n =1 is an arbitrary sequenc e satisfying lim n →∞ κ ( n ) = ∞ (12) lim n →∞ κ ( n ) β A ( n ) = 0 (13) lim n →∞ log κ ( n ) n = 0 (14) and κ denotes κ ( n ) . Since inf µ Y | X : H ( Y | X ) 0 , then the right hand side of (10) goes to zero as n → ∞ for all µ Y | X satisfying ( 11). Remark 3: I t should be noted here that we have I ( X ; Y ) > R B (15) from (11) and (9). Howe ver (11) and (15 ) do no t im ply (9) ev en wh en R A < H ( X ) . Remark 4: For β A satisfying (H3), there is κ satisfying (12)– (14) by letting κ ( n ) ≡      n ξ if β A ( n ) = o  n − ξ  1 √ β A ( n ) , otherwise (16) Novem ber 14, 2018 DRAFT 11 for ev ery n . If β A ( n ) is not o  n − ξ  , there is κ ′ > 0 such th at β A ( n ) n ξ > κ ′ and log κ ( n ) n = log 1 β A ( n ) 2 n ≤ log n ξ κ ′ 2 n = ξ log n − log κ ′ 2 n for all sufficiently large n . T his implies that κ satisfies (14). I t should be noted that we can let ξ be arbitr arily large in (1 6) when β A ( n ) vanishes expo nentially fast. This parameter ξ affects the u pper bound of (10). Remark 5: Fr om Lemm a 4, we have the fact that the con dition (H3) o f β B is n ot necessary for the ensembles ( A , p A ) and ( B , p B ) of linear functions. V I . P RO O F O F T H E O R E M S In this section, we prove the theore ms. A. Pr oof of Theorem 1 Let G U ≡ { x ′ : H ( x ′ ) ≤ H ( U ) } . If x ∈ T U and g A ( A x ) 6 = x , then there is x ′ ∈ C A ( A x ) such that x ′ 6 = x and H ( x ′ ) ≤ H ( x ) = H ( U ) , which implies that [ G U − { x } ] ∩ C A ( A x ) 6 = ∅ . Then we have E A [Erro r X ( A )] = E A " X x µ X ( x ) χ ( g A ( A x ) 6 = x ) # ≤ X U X x ∈T U µ X ( x ) p A         A : [ G U − { x } ] ∩ C A ( A x ) 6 = ∅         ≤ X U X x ∈T U µ X ( x ) max  |G U | α A | Im A| + β A , 1  ≤ X U X x ∈T U µ X ( x ) max  |X | l A 2 − n [ R − H ( U ) − λ X ] α A | Im A| , 1  + β A ≤ max  α A |X | l A | Im A| , 1  X U 2 − n [ D ( ν U k µ X )+ | R − H ( U ) | + − λ X ] + β A ≤ max  α A |X | l A | Im A| , 1  2 − n [ F X ( R ) − 2 λ X ] + β A , where th e second in equality com es from L emma 2, the th ird inequa lity comes fr om Lemma 8, the four th inequality comes from Lemmas 6 and 7, and the last ineq uality co mes from the definition of F X and Lemma 5. Then we have the fact that there is a function (matrix) A ∈ A satisfying (5). Novem ber 14, 2018 DRAFT 12 B. Pr oof of Theorem 3 Let U V ≡ ν V U be a join t type of th e sequen ce ( x , y ) ∈ X n × Y n , where th e marginal type U is d efined as U ≡ arg min U ′ D ( ν U ′ k µ X ) . (17) and the conditio nal type gi ven type U is den oted by V | U . Since R A + R B < H ( X ) and H ( U ) approache s H ( X ) as n goes to infinity be cause of the law of large numbers a nd the con tinuity of the entropy function, we have H ( U ) − λ X > R A + R B + log κ n for all sufficiently large n . Then we hav e |T U | ≥ 2 n [ H ( U ) − λ X ] , ≥ κ 2 n [ R A + R B ] = κ | Im A|| Im B | for a ll sufficiently large n , where th e first inequ ality comes f rom Lemm a 6. This im plies that there is T ⊂ T U such that κ ≤ |T | | Im A|| Im B | ≤ 2 κ (18) for all sufficiently large n . Let • g AB ( c , m ) ∈ T (UC1) • g A ( c , y ) = g AB ( c , m ) . (UC2) Then we have Erro r ( A, B , c , µ Y | X ) ≤ p M Y ( S c 1 ) + p M Y ( S 1 ∩ S c 2 ) , (19) where S i ≡ { ( m , y , w ) : (UC i ) } . First, we ev aluate E AB C [ p M Y ( S c 1 )] . W e ha ve E AB C [ p M Y ( S c 1 )] = p AB C M ( { ( A, B , c , m ) : T ∩ C AB ( c , m ) = ∅} ) ≤ α AB − 1 + | Im A|| Im B | [ β AB + 1] |T | ≤ α AB − 1 + β AB + 1 κ (20) where the equ ality comes fro m the proper ty of T , the first inequailty comes fro m Lemma 3 an d the second inequality comes from (18) . Next, we evaluate E AB C [ p M Y ( S 1 ∩ S c 2 )] . Let G ( y ) ≡ { x ′ : H ( x ′ | y ) ≤ H ( U | V ) } Novem ber 14, 2018 DRAFT 13 and assume that ( x , y ) ∈ T U V . Then we have E AC [ χ ( A x = c ) χ ( g A ( c , y ) 6 = x )] = p AC                  ( A, c ) : A x = c ∃ x ′ 6 = x s.t. H ( x ′ | y ) ≤ H ( x | y ) and A x ′ = c                  = p A         A : ∃ x ′ 6 = x s.t. H ( x ′ | y ) ≤ H ( x | y ) and A x ′ = A x         p C ( { c : A x = c } ) = 1 | Im A| p A         A : ∃ x ′ 6 = x s.t. H ( x ′ | y ) ≤ H ( U | V ) and A x ′ = A x         ≤ 1 | Im A| max    X x ′ ∈ [ G ( y ) −{ x } ] p A ( { A : A x = A x ′ } ) , 1    ≤ 1 | Im A| max  2 n [ H ( U | V )+ λ X Y ] α A | Im A| + β A , 1  = 1 | Im A| max n 2 − n [ R A − H ( U | V ) − λ X Y ] α A + β A , 1 o ≤ 1 | Im A| h max { α A , 1 } 2 − n [ | R A − H ( U | V ) | + − λ X Y ] + β A i , (21) where | · | + is defined by (2), the third equality co mes from L emma 1 an d the seco nd ineq uality comes from Lemma 8 and (H4) for an ensemble p A . Then we have E AB C [ p M Y ( S 1 ∩ S c 2 )] = E AB C M   X x ∈T X V | U X y ∈T V | U ( x ) µ Y | X ( y | x ) χ ( g AB ( c , m ) = x ) χ ( g A ( c , y ) 6 = x )   ≤ E AB C M   X x ∈T X V | U X y ∈T V | U ( x ) µ Y | X ( y | x ) χ ( A x = c ) χ ( B x = m ) χ ( g A ( c , y ) 6 = x )   = X x ∈T X V | U X y ∈T V | U ( x ) µ Y | X ( y | x ) E AC [ χ ( A x = c ) χ ( g A ( c , y ) 6 = x )] E B M [ χ ( B x = m )] ≤ 1 | Im A|| Im B | X x ∈T X V | U X y ∈T V | U ( x ) µ Y | X ( y | x ) h max { α A , 1 } 2 − n [ | R A − H ( U | V ) | + − λ X Y ] + β A i = 1 | Im A|| Im B | X x ∈T   X V | U X y ∈T V | U ( x ) µ Y | X ( y | x ) max { α A , 1 } 2 − n [ | R A − H ( U | V ) | + − λ X Y ] + β A   ≤ 1 | Im A|| Im B | X x ∈T   max { α A , 1 } X V | U 2 − n [ D ( ν V | U k µ Y | X | ν U )+ | R A − H ( U | V ) | + − λ X Y ] + β A   ≤ |T | | Im A|| Im B | h max { α A , 1 } 2 − n [ F Y | X ( R A ) − 2 λ X Y ] + β A i ≤ 2 κ h max { α A , 1 } 2 − n [ F Y | X ( R A ) − 2 λ X Y ] + β A i , (22) where the secon d in equality comes fro m Lem ma 1 and (21 ), the third inequality c omes from Lemm as 7 and 6, th e fou rth inequ ality comes fr om the definition o f F Y | X and Lemm a 5 and the last in equality comes from Novem ber 14, 2018 DRAFT 14 (18). From (19), (20), and (22) we have E AB C  Erro r Y | X ( A, B , c )  ≤ α AB − 1 + β AB + 1 κ + 2 κ h max { α A , 1 } 2 − n [ F Y | X ( R A ) − 2 λ X Y ] + β A i . Applying the ab ove a rgument for all µ Y | X satisfying (11) an d (9 ), we have the fact that there are A ∈ A , B ∈ B , and c ∈ Im A that satisfy (10). V I I . C O N C L U S I O N The fixed ra te un i versal codin g theor ems are proved by using th e no tion of hash property . W e p roved the theo rems of fixed-ra te lo ssless universal sou rce coding and fixed-rate universal chan nel coding . Since an ensemble of sparse m atrices satisfies the hash property requ irement, it is proved th at we can con struct un i versal codes by using sparse matrices. A P P E N D I X W e introduce the f ollowing lemmas that are used in the proof s o f th e th eorems. Lemma 5 ([4, Lemma 2.2]): The number o f different types of sequences in X n is fewer than [ n + 1] |X | . T he number of conditio nal typ es of sequ ences X × Y is f ewer than [ n + 1] |X || Y | . Lemma 6 ([4, Lemma 2.3]): For a type U of a sequence in X n , 2 n [ H ( U ) − λ X ] ≤ |T U | ≤ 2 nH ( U ) , where λ X is defined in (1). Lemma 7 ([4, Lemma 2.6]): 1 n log 1 µ X ( x ) = H ( ν x ) + D ( ν x k µ X ) 1 n log 1 µ Y | X ( y | x ) = H ( ν y | x | ν x ) + D ( ν y | x k µ Y | X | ν y ) . Lemma 8 ([11, Lemma 2]): For y ∈ T V , | { x ′ : H ( x ′ ) ≤ H ( U ) } | ≤ 2 n [ H ( U )+ λ X ] | { x ′ : H ( x ′ | y ) ≤ H ( U | V ) } | ≤ 2 n [ H ( U | V )+ λ X Y ] , where λ X and λ X Y are defined b y (1). Pr o of: The first ineq uality o f this lemma is shown by the second inequ ality . The secon d inequality is shown by | { x ′ : H ( x ′ | y ) < H ( U | V ) } | = X U ′ : H ( U ′ | V ) ≤ H ( U | V ) |T U ′ | V ( y ) | ≤ X U ′ : H ( U ′ | V ) ≤ H ( U | V ) 2 nH ( U ′ | V ) Novem ber 14, 2018 DRAFT 15 ≤ X U ′ : H ( U ′ | V ) ≤ H ( U | V ) 2 nH ( U | V ) ≤ [ n + 1] |X || Y | 2 nH ( U | V ) = 2 n [ H ( U | V )+ λ U V ] , where the first inequ ality com es fr om Lem ma 6 and the third inequality comes from Lemma 5. A C K N OW L E D G E M E N T S This paper was wr itten while one of au thors J. M. was a visiting r esearcher at ETH, Z ¨ urich. He wishes to thank Prof. Maurer for arran ging for his stay . R E F E R E N C E S [1] A. Bennatan and D. Burshtein, “On the applicat ion of LDPC codes to arbitra ry discrete-memoryle ss channels, ” IEEE T rans. Inform. Theory , vol. IT -50, no. 3, pp. 417–438, Mar . 2004. [2] T . P . Coleman, M. M ´ edard, and M. E f fros, “T o wards practic al miminum-entropy univ ersal decodi ng, ” Pr oc. of the IEE E Data Compr ession Cofer ence , Mar . 29–31, 2005 pp. 33–42. [3] I. Csisz ´ ar , “Linear codes for sources and source networks: Error exponen ts, univ ersal coding, ” IEEE T rans. Inform. Theory , vol. IT -28, no. 4, pp. 585–592, Jul. 1982. [4] I. Csisz ´ ar and J. K ¨ orner , Informati on Theory: Coding Theorems for Discret e Memoryless Systems , Acade mic Press, 1981. [5] J. L . Carter and M. N. W eg man, “Univ ersal classes of hash functions, ” J. Comput. Syst. Sci. , vol. 18, pp. 143–154, 1979. [6] J. Feldman, M.J. W ainwri ght, and D.R. Karger , “Using linear programming to decode binary linear codes, ” IEE E T rans. Inform. Theory , vol. IT -51, no. 3, pp. 954–972, Mar . 2005. [7] H. Koga, “Source coding using familie s of uni versal hash functions, ” IEEE T rans. Inform. Theory , vol. IT -53, no. 9, pp. 3226–3233, Sept. 2007. [8] S. Miyake and M. Maruya ma, “Constructi on of univ ersal codes using L DPC matrices and their error exponents, ” IEICE T rans. Fundamental s , vol. E90-A, No. 9, pp. 1830–1839, Sept. 2007. [9] S. Miyake and J. Muramatsu, “ A construction of channel code, JSCC and univ ersal code for discret e memoryless channel s using sparse matrice s, ” to appear in Pro c. 2008 IEEE Int. Symp. Inform. Theory , Tronto, Canada, Jul. 6–11, 2008. [10] J. Muramatsu, T . Uyematsu, and T . W adayama, “Low density parity check matric es for coding of correlated sources, ” IE EE T rans. Inform. Theory , vol. IT -51, no. 10, pp. 3645–3653, Oct. 2005. [11] J. Muramatsu, “Secret key agreement from correlat ed source outputs using low density parit y check matrices, ” IEICE T rans. Fundamental s , vol. E89-A, no. 7, pp. 2036–2046, Jul. 2006. [12] J. Muramatsu and S. Miyake “Hash property and coding theorems for s parse matric es and maximal-lik elihoo d coding, ” submittd to IEEE T rans. Inform. Theory , av aila ble at arXiv:0801.387 8 [cs.IT] , 2007. Novem ber 14, 2018 DRAFT