Entropy rate is a real valued functional on the space of discrete random sources which lacks a closed formula even for subclasses of sources which have intuitive parameterizations. A good way to overcome this problem is to examine its analytic properties relative to some reasonable topology. A canonical choice of a topology is that of the norm of total variation as it immediately arises with the idea of a discrete random source as a probability measure on sequence space. It is shown that entropy rate is Lipschitzian relative to this topology, which, by well known facts, is close to differentiability. An application of this theorem leads to a simple and elementary proof of the existence of entropy rate of random sources with finite evolution dimension. This class of sources encompasses arbitrary hidden Markov sources and quantum random walks.
Entropy rate is a key quantity in information theory as it is equal to the average amount of information per symbol of discrete-time, discrete-valued stochastic processes (usually referred to as discrete random sources in the following). Therefore, it is natural to ask how entropy rate behaves if knowledge of discrete random sources is subject to uncertainties which, for example, may be inherent to inference processes and/or originate from noisy channels. However, closed formulas for entropy rate exist only for rare examples of classes of discrete random sources. For instance, already hidden Markov sources (HMSs) seem to defy a convenient formula although there is one for the special case of Markov sources. Therefore, in this case, recent efforts focused on the direct investigation of analytic properties of entropy rate like smoothness or even analyticity [20,21], [31,30], [25], [18].
The purpose of this paper is to contribute to the issue of analytic properties of entropy rate in a more general fashion. Namely, we study the behavior of entropy rate relative to the topology induced by the norm of total variation. This topology is one of the natural choices and it is ubiquitous in both theoretical and practical work. We show that entropy rate is Lipschitzian on the whole space of discrete random sources which is, due to an elementary theorem of Rademacher, close to differentiability.
We will use this result to give an elementary proof of the existence of entropy rate for sources with finite evolution dimension [6] which contain the classes of arbitrary HMSs [24] and quantum random walks (QRWs) [1], [5].
The paper is organized as follows. We will identify discrete random sources with probability measures acting on the measurable space of symbol sequences equipped with the σ-algebra generated by the cylinder sets of sequences. Therefore, in section 2, we will briefly compile the theory’s standard arguments. In section 3 we prove that entropy rate is Lipschitz continuous relative to the topology induced by the norm of total variation which is the main contribution of this paper. In section 4 we demonstrate how to exploit this result for an elementary proof of existence of random sources with finite evolution dimension which include HMSs and QRWs as special cases. In section 5 we will describe the proof’s intuition thereby commenting on open problems such as other choices of topology and/or stricter choices of analytic properties.
As usual, Σ * = ∪ t≥0 Σ t is the set of all words (strings of finite length) over the finite alphabet Σ together with the concatenation operation
Throughout this paper
Σ is the set of sequences over Σ and B is the σ-algebra generated by the cylinder sets. Cylinder sets B are identified with sets of words A B ⊂ Σ t such that B is the set of sequences which start with the words in A B . In general, the cardinality of a set A is denoted by |A|.
We view stochastic processes (X t ) t∈N with values in Σ as probability measures P X on the measurable space (Ω, B) and vice versa via the relationship (v = v 0 …v t-1 ∈ Σ t corresponds to the cylinder set of sequences having v as prefix)
where the term on the right hand side is the probability that the random source emits the symbols v 0 , …, v t-1 at periods 0, …, t -1. Note that a stochastic process (X t ) is uniquely determined by the values P X (v) for all v ∈ Σ * as the cylinder sets corresponding to words v generate B
Although being a canonical choice of norm (see appendix A for a short review of the related theory and corresponding definitions), computation of the norm of total variation would not be easy for the measurable space under consideration by means of its original definition alone. The following lemma shows a concrete way to get a grip of the corresponding topology. Exact definition and basic properties of the norm of total variation have been deferred to appendix A.
where P X , P Y are probability measures associated to discrete random sources (X t ), (Y t ).
Proof. See sec. A.2 of the appendix for the predominantly measure theoretical arguments. ⋄
In the following, we will refer to the quantities
as upper entropy rate resp. lower entropy rate of a random source (X t ) with associated probability measure P X , where, using the language introduced above,
is the entropy of the distribution over the words of length t induced by the random source, divided by t. Entropy rate of a random source (X t ) with associated probability measure P X is denoted by
The existence of the limit of the H t (P X ) is also referred to as the existence of entropy rate where, obviously, a necessary and sufficient condition for entropy rate to exist is
Throughout this paper,
is the usual regular n -1-dimensional simplex in R n and, for technical convencience, log is the natural logarithm. Note that, as it is more common to use the logarithm to the base 2, switching bases does not affect any analytic property of entropy rate.
Our ma
This content is AI-processed based on open access ArXiv data.
