Joint Source-Channel Coding Revisited: Information-Spectrum Approach

Typically, the traditional separation theorem, which we call the separation theorem in the narrow sense, states that if the infimum R f (V) of all achievable fixed-length coding rates for the source V is smaller than the capacity C(W) for the channel W, then the source V is reliably transmissible by two step encoding over the channel W; whereas if R f (V) is larger than C(W) then the reliable transmission is impossible. While the former statement is always true for any general source V and any general channel W, the latter statement is not always true. Then, a very natural question may be raised for what class of sources and channels and in what sense the separation theorem holds in general.

Shannon [1] has first shown that the separation theorem holds for the class of stationary memoryless sources and channels. Since then, this theorem has received extensive attention by a number of researchers who have attempted to prove versions that apply to more and more general classes of sources and channels. Among others, for example, Dobrushin [4], Pinsker [5], and Hu [6] have studied the separation theorem problem in the framework of information-stable sources and channels.

Recently, on the other hand, Vembu, Verdú and Steinberg [9] have put forth this problem in a much more general information-spectrum context with general source V and general channel W. From the viewpoint of information spectra, they have generalized the notion of separation theorem and shown that, usually in many cases even with R f (V) > C(W), it is possible to reliably transmit the output of the source V over the channel W. Furthermore, in terms of information spectra, they have established a sufficient condition for the transmissibility as well as a necessary condition. It should be noticed here that, in this general joint source-channel coding situation, what indeed matters is not the validity problem of the traditional type of separation theorems but the derivation problem of necessary and/or sufficient conditions for the transmissibility from the information-spectrum point of view.

However, while their sufficient condition looks simple and significantly tight, their necessary condition does not look quite close to tight.

The present paper was mainly motivated by the reasonable question why the forms of these two conditions look rather very different from one another. First, in Section 3, the basic tools to answer this question are established, i.e., two fundamental lemmas: a generalization of Feinstein’s lemma [2] and a generalization of Verdú-Han’s lemma [8], which provide with the very basis for the key results to be stated in the subsequent sections. These lemmas are of dualistic information-spectrum forms, which is in nice accordance with the general joint source-channel coding framework. In Section 4, given a general source V and a general channel W, we establish, in terms of information-spectra, a sufficient condition (Direct theorem) for the transmissibility as well as a necessary condition (Converse theorem). The forms of these two conditions are very close from each other, and “fairly” coincides with one another, provided that we dare disregard some relevant asymptotically vanishing term.

Next, we equivalently rewrite these conditions in the forms useful to see relations to the separation theorem. As a consequence, it turns out that a separation-theorem-like equivalent of our sufficient condition just coincides with the sufficient condition given by Vembu, Verdú and Steinberg [9], whereas a separation-theorem-like equivalent of our necessary condition is shown to be strictly stronger than or equivalent to the necessary condition given by them. Here it is pleasing to observe that a nice duality is found between our necessary and sufficient conditions, whereas we cannot fully enjoy such a duality between the necessary condition and the sufficient condition by Vembu, Verdú and Steinberg [9].

On the other hand, in Section 5, we demonstrate a sufficient condition as well as a necessary condition for the ε-transmissibility, which is the generalization of the sufficient condition as well as the necessary condition as was shown in Section 4. Finally, in Section 6, we restrict the class of sources and channels to those that satisfy the strong converse property (or, more generally, the semi-strong converse property) to show that the separation theorem in the traditional sense holds for this class.

In this preliminary section, we prepare the basic notation and definitions which will be used in the subsequent sections.

Let us first give here the formal defintion of the general source. A general sources is defined as an infinite sequence

values in a countably infinite set V that we call the source alphabet. It should be noted here that each component of V n may change depending on block length n. This implies that the sequence V is quite general in the sense that it may not satisfy even the consistency condition as usual processes, where the consistency condition means that for any integers m, n such that m < n it holds that

The class of sources thus defined covers a very wide range of sources including all nonstationary and/or nonergodic sources (cf. Han and Verdú [7]).

The formal definition of a general channel is as follows. Let X , Y be arbitrary abstract (not necessarily countable) sets, which we call the input alphabet and the output alphabet, respectively. A general channel is defined as an infinite sequence W = {W n : X n → Y n } ∞ n=1 of n-dimensional probability transition matrices W n , where W n (y|x) (x ∈ X n , y ∈ Y n ) denotes the conditonal probability of y given x. * The class of channels thus defined covers a very wide range of channels including all nonstationary and/or nonergodic channels with arbitrary memory structures (cf. Han and Verdú [7]).

Remark 2.1 A more reasonable definition of a general source is the following. Let {V n } ∞ n=1 be any sequence of arbitrary source alphabets V n (a countabley infinite or abstract set) and let V n be any random variable taking values in

Verdú and Han [10]). The above definition is a special case of this general source with

On the other hand, a more reasonable definition of the general channel is the following. Let {W n : X n → Y n } ∞ n=1 be any sequence of arbitrary probability transition matrices, where X n , Y n are arbitrary abstract sets. Then, the sequence W = {W n } ∞ n=1 of probability transition matrices W n is called a general channel (cf. Han [11]). The above definition is a special case of this general channel with

The results in this paper (Lemma 3.1, Lemma 3.2, Theorem 4.1, Theorem 4.2, Theorem 4.3, Theorem 4.4, Theorem 5.1, Theorem 5.2 and Theorems 6.1 ∼ 6.7 ) continue to be valid as well also in this more general setting with

In the sequel we use the convention that P Z (•) denotes the probability distribution of a random variable Z, whereas P Z|U (•|•) denotes the conditional probability distribution of a random variable Z given a random variable U . ✷

n )} ∞ n=1 be any general source, and let

n=1 be any general channel. We consider an encoder ϕ n : V n → X n and a decoder ψ n : Y n → V n , and put X n = ϕ n (V n ). Then, denoting by Y n the output from the channel W n due to the input X n , we have the obvious relation:

(2.1)

The error probability ε n with code (ϕ n , ψ n ) is defined by

where D(v) ≡ {y ∈ Y n |ψ n (y) = v} (∀v ∈ V n ) (D(v) is called the decoding set for v) and “c” denotes the complement of a set. A pair (ϕ n , ψ n ) with error probability ε n is simply called a joint source-channel code (n, ε n ). We now define the transmissibility in terms of joint source-channel codes (n, ε n ) as

With this definition of transmissibility, in the following sections we shall establish a sufficient condition as well as a necessary condition for the transmissibility when we are given a geneal source V and a general channel W. These two conditions are very close to each other and could actually be seen as giving “almost the same condition,” provided that we dare disregard an asymptotically negligible term γ n → 0 appearing in those conditions (cf. Section 4).

The quantity ε n defined by (2.2) is more specifically called the average error probability, because it is averaged with respect to P V n (v) over all source outputs v ∈ V n . On the other hand, we may define another kind of error probability by

which we call the maximum error probability. It is evident that the transmissibility in the maximum sense implies the transmissibility in the average sense. However, the inverse is not necessarily true. To see this, it suffices to consider the following simple example. Let the source, channel input, channel output alphabets be V n = {0, 1, 2}, X n = {1, 2}, Y n = {1, 2}, respectively; and the (deterministic) channel W n : X n → Y n be defined by W n (j|i) = 1 for i = j, W n (1|0) = 1. Moreover, let the source V n have probability distribution P Vn (0) = α n , P Vn (1) = P Vn (2) = 1-αn 2 (α n → 0 as n → ∞). One of the best choices of possible pairs of encoder-decoder (ϕ n : V n → X n , ψ n : Y n → V n ), either in the average sense or in the maximum sense, is such that ϕ n (i) = i for i = 1, 2; ϕ n (0) = 1; ψ n (i) = i for i = 1, 2. Then, the average error probability is ε a n = α n → 0, while the maximum error probability is ε m n = 1. Thus, in this case, the source V n is transmissible in the average sense over the channel W n , while it is not transmissible in the maximum sense.

Hereafter, the probability ε n is understood to denote the “average” error probability, unless otherwise stated. ✷

In this section, we prepare two fundamental lemmas that are needed in the next section in order to establish the main theorems (Direct part and Converse part).

Lemma 3.1 (Generalization of Feinstein’s lemma) Given a general source

, let X n be any input random variable taking values in X n and Y n be the channel output via W n due to the channel input X n , where

where † γ > 0 is an arbitrary positive number.

Remark 3.1 In a special case where the source

which implies that the entropy spectrum ‡ of the source V = {V n } ∞ n=1 is exactly one point spectrum concentrated on 1 n log M n . Therefore, in this special case, Lemma 3.1 reduecs to Feinstein’s lemma [2]. ✷

For each v ∈ V n , generate x(v) ∈ X n at random according to the conditional distribution P X n |V n (•|v) and let x(v) be the codeword for v. In other words, we define the encoder ϕ n : V n → X n as ϕ n (v) = x(v), where † In the case where the input and output alphabets X , Y are abstract dy) is the Radon-Nikodym derivative that is measurable in (x, y).

‡ The probablity distribution of 1 n log

Han and Verdú [7]).

{x(v) | ∀v ∈ V n } are all independently generated. We define the decoder ψ n : Y n → V n as follows: Set

where for simplicity we have put

Suppose that the decoder ψ n received a channel output y ∈ Y n . If there exists one and only one v ∈ V n such that (x(v), y) ∈ S n (v), define the decoder as ψ n (y) = v; otherwise, let the output of the decoder ψ n (y) ∈ V n be arbitrary. Then, the probability ε n of error for this pair (ϕ n , ψ n ) (averaged over all the realizatioins of the random code) is given by

where ε n (v) is the probability of error (averaged over all the realizatioins of the random code) when v ∈ V n is the source output. We can evaluate ε n (v) as

where Y n is the channel output via W n due to the channel input x(v). The first term on the right-hand side of (3.5) is written as

Hence,

On the other hand, noting that x(v ′ ), x(v) (v ′ = v) are independent and hence x(v ′ ), Y n are also independent, the second term on the right-hand side of (3.5) is evaluated as

Hence,

On the other hand, in view of (3.2), (3.3), (x, y) ∈ S n (v ′ ) implies

Therefore, (3.7) is further transformed to

Then, from (3.4), (3.6) and (3.8) it follows that

Thus, there must exist a deterministic (n, ε n ) code such that

thereby proving Lemma 3.1.

and W = {W n } ∞ n=1 be a general source and a general channel, respectively, and let ϕ n : V n → X n be the encoder of an (n,

where γ > 0 is an arbitrary positive number.

Remark 3.2 In a special case where the source

which implies that the entropy spectrum of the source V = {V n } ∞ n=1 is exactly one point spectrum concentrated on 1 n log M n . Therefore, in this special case, Lemma 3.2 reduecs to Verdú-Han’s lemma [8].

✷

and, for each v ∈ V n set

Then, noting the Markov chain property (2.1), we have

W n (y|x), (

where we have used the relation:

Now, it follows from (3.10) and (3.11) that y ∈ B(v, x) implies

which is substituted into the right-hand side of (3.12) to yield

thereby proving the claim of the lemma.

In this section we give both of a sufficient condition and a necessary condition for the transmissibility with a given general souce V = {V n } ∞ n=1 and a given general channel W = {W n } ∞ n=1 . First, Lemma 3.1 immediately leads us to the following direct theorem:

n=1 be a general source and a general channel, respectively. If there exist some channel input

for which it holds that

then the source

, where Y n is the channel output via W n due to the channel input X n and V n → X n → Y n .

Since in Lemma 3.1 we can choose the constant γ > 0 so as to depend on n, let us take, instead of γ, an arbitrary {γ n } ∞ n=1 satisfying condition (4.1). Then, the second term on the right-hand side of (3.1) vanishes as n tends to ∞, and hence it follows from (4.2) that the right-hand side of (3.1) vanishes as n tends to ∞. Therefore, the (n, ε n ) code as specified in Lemma 3.1 satisfies lim n→∞ ε n = 0. ✷ Next, Lemma 3.2 immediately leads us to the following converse theorem:

n=1 where ϕ n : V n → X n is the channel encoder. Then, for any sequence {γ n } ∞ n=1 satisfying condition (4.1), it holds that

where Y n is the channel output via W n due to the channel input X n and

Proof:

for simplicity, (4.2) is written as

which can be transformed to

then by virtue of (4.4) and Markov inequality, we have

Let us now define the upper cumulative probabilities for A n , B n by

then it follows that

u≥t-γn

u≥t-γn

On the other hand, by means of (4.5), u ∈ T n implies that

Theore, by (4.6), (4.7) it is concluded that

u≥t-γn

That is,

This means that, for all t, the upper cumulative probability P n (t) of A n is larger than or equal to the upper cumulative probability Q n (t -γ n ) of B n , except for the asymptotically vanishing difference 2 √ α n . This in turn implies that, as a whole, the mutual information spectrum of the channel is shifted to the right in comparison with the entropy spectrum of the source. With -γ n instead of γ n , the same implication follows also from (4.3). It is such an allocation relation between the mutual information spectrum and the entropy spectrum that enables us to make an transmissible joint sourcechannel coding.

However, it is not easy in general to check whether conditions (4.2), (4.3) in these forms are satisfied or not. Therefore, we consider to equivalently rewrite conditions (4.2), (4.3) into alternative information-spectrum forms hopefully easier to depict an intuitive picture. This can actually be done by re-choosing the input and output variables X n , Y n as below. These forms are useful in order to see the relation of conditions (4.2), (4.3) with the so-called separation theorem.

First, we show another information-spectrum form equivalent to the sufficient condition (4.2) in Theorem 4.1.

The following two conditions are equivalent: 1) For some channel input X = {X n } ∞ n=1 and some sequence {γ n } ∞ n=1 satisfying condition (4.1), it holds that

where Y n is the channel output via W n due to the channel input X n and

2. (Strict domination: Vembu, Verdú and Steinberg [9]) For some channel input X = {X n } ∞ n=1 , some sequence {c n } ∞ n=1 and some sequence {γ n } ∞ n=1 satisfying condition (4.1), it holds that

where Y n is the channel output via W n due to the channel input X n .

Remark 4.2 (separation in general) @ The sufficient condition 2) in Theorem 4.3 means that the entropy spectrum of the source and the mutual information spectrum of the channel are asymptotically completely split with a vacant boundary of asymptotically vanishing width γ n , and the former is placed to the left of the latter, where these two spectra may oscillate “synchronously” with n. In the case where such a separation condition 2) is satisfied, we can split reliable joint source-channel coding in two steps as follows (separation of source coding and channel coding): We first encode the source output V n at the fixed-length coding rate c n = 1 n log M n (M n is the size of the message set M n ), and then encode the output of the source encoder into the channel. The error probabilty ε n for this two step coding is upper bounded by the sum of the error probability of the fixed-length source coding (cf. Vembu, Verdú and Steinberg [9]; Han [11, Lemma 1.3.1]):

and the “maximum” error probability of the channel coding (cf. Feinstein [2], Ash [3], Han [11, Lemma 3.4.1]):

It then follows from (4.9) that both of these two error probabilities vanish as n tends to ∞, where it should be noted that e -nγn → 0 as n → ∞. Thus, we have lim

n=1 . This can be regarded as providing another proof of Theorem 4.1. ✷

Proof of Theorem 4.3:

2. ⇒ 1): For any joint probability distribution P V n X n for V n and X n , we have

which together with (4.9) implies (4.8).

1. ⇒ 2)F Supposing that condition 1) holds, put

and moreover, with

Furthermore, define

)

then the joint probability distribution P V n X n Y n can be written as a mixture:

where

, respectively. We notice here that the Markov chain property

We now rewrite (4.10) as

On the other hand, since (4.11), (4.12) lead to λ

(1)

Then, by the definition of Ṽ n ,

and so from (4.16), we obtain

Next, since it follows from (4.14) that P Y n (y) = λ (1) n P Ỹ n (y) + λ (2) n P Y n (y) ≥ λ (1) n P Ỹ n (y)

we have

which is substituted into (4.17) to get

On the other hand, by the definition (4.11

Finally, resetting Xn Ỹ n , 1 4 γ n as X n Y n and γ n , respectively, we conclude that condition 2), i.e., (4.9) holds. ✷

Having established an information-spectrum separation-like form of the sufficient condition (4.2) in Theorem 4.1, let us now turn to demonstrate several information-spectrum versions derived from the necessary condition (4.3) in Theorem 4.2.

where Y n is the channel output via W n due to the channel input X n and

2. For any sequence {γ n } ∞ n=1 satisfying condition (4.1) and for some channel input

where Y n is the channel output via W n due to the channel input X n and

Proof: The necessity of condition 1) immediately follows from necessity condition (4.3) in Theorem 4.2. Moreover, it is also trivial to see that condition 1) implies condition 2) as an immediate logical consequence, and hence condition 2) is also a necessary condition. ✷

The necessary condition 1) in Theorem 4.4 below is the same as condition 2) in Proposition 4.1. This is written here again in order to emphasize a pleasing duality between Theorem 4.3 and Theorem 4.4, which reflects on the duality between two fundamental Lemmas 3.1 and 3.2 .

The following two conditions are equivalent: 1) For any sequence {γ n } ∞ n=1 satisfying condition (4.1) and for some channel input

where Y n is the channel output via W n due to the channel input X n and

2. (Domination) For any sequence {γ n } ∞ n=1 satisfying condition (4.1) and for some channel input X = {X n } ∞ n=1 and some sequence {c n } ∞ n=1 , it holds that

where Y n is the channel output via W n due to the channel input X n .

This theorem can be proved in the entirely same manner as in the proof of Theorem 4.3 with γ n replaced by -γ n . ✷ Remark 4.3 Originally, the definition of domination given by Vembu, Verdú and Steinberg [9] is not condition 2) in Theorem 4.4 but the following:

2 ′ ) (Domination) For any sequence {d n } ∞ n=1 and any sequence {γ n } ∞ n=1 satisfying condition (4.1), there exists some channel input

holds, where Y n is the channel output via W n due to the channel input X n . ✷ This necessary condition 2 ′ ) is implied by necessary condition 2) in Theorem 4.4. To see this, set

Then, we observe that κ n ≤ α n if d n ≥ c n ; and µ n ≤ β n if d n ≤ c n , and hence it follows from condition 2) that κ n µ n ≤ α n +β n → 0 as n tends to ∞. Thus, condition 2) implies condition 2 ′ ), which means that condition 2) is strictly stronger than or equivalent to condition 2 ′ ) as necessary conditions for the transmissibility. It is not currently clear, however, whether both are equivalent or not. ✷ Remark 4.4 Condition 2) in Theorem 4.4 of this form is used later to directly prove Theorem 6.6 (separation theorem), while condition 2 ′ ) in Remark 4.3 of this form is irrelevant for this purpose. ✷

So far we have considered only the case where the error probability ε n satisfies the condition lim n→∞ ε n = 0. However, we can relax this condition as follows:

lim sup

where ε is any constant such that 0 ≤ ε < 1. (It is obvious that the special case with ε = 0 coincides with the case that we have considered so far.) We now say that the source V is ε-transmissible over the channel W when there exists an (n, ε n ) code satisfying condition (5.1).

Then, the same arguments as in the previous sections with due slight modifications lead to the following two theorems in parallel with Theorem 4.1 and Theorem 4.2, respectively:

be a general source and a general channel, respectively. If there exist some channel input X = {X n } ∞ n=1 and some sequence {γ n } ∞ n=1 such that

for which it holds that lim sup

, where Y n is the channel output via W n due to the channel input

, and let the channel input be

n=1 where ϕ n : V n → X n is the channel encoder. Then, for any sequence {γ n } ∞ n=1 satisfying condition (5.2), it holds that lim sup

where Y n is the channel output via W n due to the channel input X n and

It should be noted here that such a sufficient condition (5.3) as well as such a necessary condition (5.4) for the ε-transmissibility cannot actually be derived in the way of generalizing the strict domination in (4.9) and the domination in (4.23). It should be noted also that, under the ε-transmissibility criterion, joint source-channel coding is beyond the separation principle. ✷

Thus far we have investigated the joint source-channel coding problem from the viewpoint of information spectra and established the fundamental theorems (Theorems 4.1∼4.4). These results are of seemingly different forms from separation theorems of the traditional type. Then, it would be natural to ask a question how the separation principle of the information spectrum type is related to separation theorems of the traditional type. In this section we address this question.

To do so, we first need some preparation. We denote by R f (V) the infimum of all achievable fixed-length coding rates for a general source V = {V n } ∞ n=1 (as for the formal definition, see Han and Verdú [7], Han [11, Definitions 1.1.1, 1.1.2]), and denote by C(W) the capacity of a general channel W = {W n : X n → Y n } ∞ n=1 (as for the formal definition, see Han and Verdú [7], Han [11,Definitions 3.1.1,3.1.2]). First, R f (V) is characterized as Theorem 6.1 (Han and Verdú [7], Han [11])

where

Next, let us consider about the characterization of C(W). Given a general channel W = {W n } ∞ n=1 and its input

Then, the capacity C(W) is characterized as follows.

Theorem 6.2 (Verdú and Han [8], Han [11])

where sup X means the supremum over all possible inputs X. ✷ With these preparations, let us turn to the separation theorem problem of the traditional type. A general source V = {V n } ∞ n=1 is said to be information-stable (cf. Dobrushin [4], Pinsker [5]) if

where

) and H(V n ) stands for the entropy of V n (cf. Cover and Thomas [13]). Moreover, a general channel W = {W n } ∞ n=1 is § For an arbitrary sequence of real-valued random variables {Zn} ∞ n=1 , we define the following notions (cf. Han and Verdú [7], Han [11]): p-lim sup n→∞ Zn ≡ inf{α | limn→∞ Pr {Zn > α} = 0} (the limit superior in probability), and p-lim infn→∞ Zn ≡ sup{β | limn→∞ Pr {Zn < β} = 0} (the limit inferior in probability).

said to be information-stable (cf. Dobrushin [4], Pinsker [5], Hu [6]) if there exists a channel input X = {X n } ∞ n=1 such that

where

and Y n is the channel output via W n due to the channel input X n ; and I(X n ; Y n ) is the mutual information between X n and Y n (cf. Cover and Thomas [13]). Then, we can summarize a typical separation theorem of the traditional type as follows.

Theorem 6.3 (Dobrushin [4], Pinsker [5]) Let the channel W = {W n } ∞ n=1 be information-stable and suppose that the limit lim n→∞ C n (W n ) exists, or, let the source V = {V n } ∞ n=1 be information-stable and suppose that the limit lim n→∞ H n (V n ) exists. Then, the following two statements hold:

In this case, we can separate the source coding and the channel coding.

2. If the source V is transmissible over the channel W, then it must hold that R f (V) ≤ C(W). ✷

In order to generalize Theorem 6.3, we need to introduce the concept of optimistic coding. The “optimistic” standpoint means that we evaluate the coding reliability with error probability lim inf n→∞ ε n = 0 (that is, ε n < ∀ε for infinitely many n). In contrast with this, the standpoint that we have taken so far is called pessimistic with error probability lim n→∞ ε n = 0 (that is, ε n < ∀ε for all sufficiently large n).

The following one concerns the optimistic source coding with any general source V.

Then, for any general source V = {V n } ∞ n=1 we have: Theorem 6.4 (Chen and Alajaji [14])

On the other hand, the next one concerns the optimistic channel capacity. Then, with a general channel W = {W n } ∞ n=1 we have Theorem 6.5 (Chen and Alajaji [14])

where Y n is the output due to the input X = {X n } ∞ n=1 . ✷ Remark 6.1 It is not difficult to check that, in parallel with Theorem 6.4 and Theorem 6.5, Theorem 6.1 and Theorem 6.2 can be rewritten as

from which, together with Theorem 6.4 and Theorem 6.5, it immediately follows that

Now, we have:

be a general source. Then, the following two statements hold:

1. If R f (V) < C(W), then the source V is transmissible over the channel W. In this case, we can separate the source coding and the channel coding.

2. If the source V is transmissible over the channel W, then it must hold that

R f (V) ≤ C(W). with, for example, c n = 1 2 (R f (V) + C(W)). Therefore, the source V is transmissible over the channel W.

2): If the source V is transmissible over the channel W, then condition 2) in Theorem 4. 4 On the other hand, (6.18) implies that H(V) ≤ lim sup n→∞ c n . Furthermore, (6.20) together with Theorem 6.5 gives us

Finally, note that R f (V) = H(V) by Theorem 6.1. ✷

We are now interested in the problem of what conditions are needed to attain equalities R f (V) = R f (V) and/or C(W) = C(W) in Theorem 6.6 and so on. To see this, we need the following four definitions: Definition 6.6 A general source V = {V n } ∞ n=1 is said to satisfy the strong converse property if H(V) = H(V) holds (as for the operational meaning, refer to Han [11]), where where Y n is the channel output via W n due to the channel input X n . ✷ With these definitions, we have the following lemmas: Lemma 6.1

1. The information-stability of a source V (resp. a channel W) with the limit implies the strong converse property of V (resp. W).

✷Now, let us think of the implication of conditions (4.2) and (4.3). First, let us think of (4.2). Putting

✷

f (V) ≤ lim inf

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11], Verdú and Han[8]), whereI(X; Y) = p-lim sup n→∞ 1 n log W n (Y n |X n ) P Y n (Y n ) . X I(X; Y),(6.23)

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11], Verdú and Han[8]), whereI(X; Y) = p-lim sup n→∞ 1 n log W n (Y n |X n ) P Y n (Y n ) . X I(X; Y),

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11], Verdú and Han[8]), where

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11], Verdú and Han[8]

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11], Verdú and Han

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han[11]

X I(X; Y) = sup X I(X; Y) (6.21)holds (as for the operational meaning, refer to Han

X I(X; Y) = sup X I(X; Y) (6.21)

• In the case where the output alphabet Y is abstract, W n (y|x) is understood to be the (conditional) probability measure element W n (dy|x) that is measurable in x.