Asymptotic Equipartition Property of Output when Rate is above Capacity

In this paper, we are interested in the case when the rate is above capacity. We will show that such a pattern that can be used for decoding will disappear when there are too many input sequences, i.e., when the rate is above capacity. Instead, in this case, the output will have an asymptotic equipartition property on the set of typical output sequences (typical with respect to p 0 (y) = x p 0 (x)p(y|x)). Interestingly, this set is independent of the specific codebook used, as long as the codebook is typical according to the random codebook generation. The reason for this equipartition is that the input sequences are too dense, so that different input sequences can contribute to the same output sequence and get mixed up.

Part of the work [1] was presented at CWIT 2009.

Investigating the optimal compress-and-forward relay scheme has motivated this study of output distribution when rate is above capacity. The optimality of the compress-and-forward schemes is arguably one of the most critical problems in the development of network information theory, where ambiguity always arises when decoding cannot be done correctly. In the classical approach of [2], the compression scheme at the relay was only based on the distribution used for generating the codebook at the source, instead of the specific codebook generated. While many different codebooks can be generated according to the same distribution, can the knowledge of the specific codebook be helpful? There have been some discussions on this issue (e.g., [3]). Here, in this paper, we show that the observations at the relay are somehow independent of the specific codebook used at the source, and only depend on the distribution by which the codebook is generated.

To further explore the optimality of the compress-and-forward schemes, we compare the rates needed to losslessly compress the relay’s observation in two different scenarios: i) the relay uses the knowledge of the source’s codebook to do the compression; ii) the relay simply ignores this knowledge. It is shown that the minimum required rates in both scenarios are the same when the rate of the source’s codebook is above the capacity of the source-to-relay link.

The remainder of the paper is organized as the following. In Section II, we first introduce some standard definitions of strongly typical sequences, and then give a definition of typical codebooks. Then, we summarize our main results in Section III, followed by the proof of these results in Section IV, V and VI. Finally, as an application of the results, the optimality of the compress-and-forward schemes is discussed in Section VII.

Consider a discrete memoryless channel (X , p(y|x), Y) with capacity C := max p(x) I(X; Y ).

Under the random coding framework, a random codebook C with respect to p 0 (x) with rate R and block length n is defined as

where each codeword in C is an i.i.d. random sequence generated according to a fixed input distribution p 0 (x).

It is well known that information can be transmitted with arbitrarily small probability of error for sufficiently large n if R < C. In this paper, however, we are interested in the case where the rate is above capacity.

We begin with some standard definitions on strong typicality [3,Ch.13].

Definition 2.1: The ǫ-strongly typical set with respect to p 0 (x), denoted by A (n) ǫ,0 (X), is the set of sequences x n ∈ X n satisfying:

1. For all a ∈ X with p 0 (a) > 0,

Similarly, we can define the ǫ-strongly typical set with respect to p 0 (y) and denote it by

Definition 2.2: The ǫ-strongly typical set with respect to p 0 (x, y), denoted by

N(a, b|x n , y n ) is the number of occurrences of the pair (a, b) in the pair of sequences (x n , y n ).

The ǫ-strongly conditionally typical set with the sequence x n with respect to the conditional distribution p(y|x), denoted by

2. For all (a, b) ∈ X × Y with p(b|a) = 0,

Definition 2.4: For the discrete memoryless channel (X , p(y|x), Y), the channel noise is said to be ǫ-typical if for any given input x n , the output Y n is ǫ-strongly conditionally typical with

x n with respect to the channel transition function p(y|x), i.e., Y n ∈ A

Due to the Law of Large Numbers, the channel noise is “typical” with high probability.

ǫ,0 , where

ǫ,0 (Y )|. Consider the set F ǫ,0 (i) ⊆ X n , where each sequence in F ǫ,0 (i) is strongly typical and can reach y n ǫ,0 (i) over a channel with typical noise, i.e.,

The following notation is useful for defining the typical codebooks.

where Xn is drawn i.i.d. according to p 0 (x) and I(A) is the indicator function:

is said to be ǫ-typical with respect to p 0 (x) if

The main results of this paper

…(Full text truncated)…