In this paper we formalize the notions of information elements and information lattices, first proposed by Shannon. Exploiting this formalization, we identify a comprehensive parallelism between information lattices and subgroup lattices. Qualitatively, we demonstrate isomorphisms between information lattices and subgroup lattices. Quantitatively, we establish a decisive approximation relation between the entropy structures of information lattices and the log-index structures of the corresponding subgroup lattices. This approximation extends the approximation for joint entropies carried out previously by Chan and Yeung. As a consequence of our approximation result, we show that any continuous law holds in general for the entropies of information elements if and only if the same law holds in general for the log-indices of subgroups. As an application, by constructing subgroup counterexamples we find surprisingly that common information, unlike joint information, obeys neither the submodularity nor the supermodularity law. We emphasize that the notion of information elements is conceptually significant--formalizing it helps to reveal the deep connection between information theory and group theory. The parallelism established in this paper admits an appealing group-action explanation and provides useful insights into the intrinsic structure among information elements from a group-theoretic perspective.
Information theory was born with the celebrated entropy formula measuring the amount of information for the purpose of communication. However, a suitable mathematical model for information itself remained elusive over the last sixty years. It is reasonable to assume that information theorists have had certain intuitive conceptions of information, but in this paper we seek a mathematic model for such a conception. In particular, building on Shannon's work [1], we formalize the notion of information elements to capture the syntactical essence of information, and identify information elements with σ-algebras and sample-space-partitions.
As we shall see in the following, by building such a mathematical model for information and identifying the lattice structure among information elements, the seemingly surprising connection between information theory and group theory, established by Chan and Yeung [2], is revealed via isomorphism relations between information lattices and subgroup lattices. Consequently, a fully-fledged and decisive approximation relation between the entropy structure of information lattices and the subgroup-index structure of corresponding subgroup lattices is obtained.
We first motivate our formal definition for the notion of information elements.
Recall the profound insight offered by Shannon [3] on the essence of communication: “the fundamental problem of communication is that of reproducing at one point exactly or approximately a message selected at another point.” Consider the following motivating example. Suppose a message, in English, is delivered from person A to person B. Then, the message is translated and delivered in German by person B to person C (perhaps because person C does not know English). Assuming the translation is faithful, person C should receive the message that person A intends to convey. Reflecting upon this example, we see that the message (information) assumes two different “representations” over the process of the entire communication-one in English and the other in German, but the message (information) itself remains the same. Similarly, coders (decoders), essential components of communication systems, perform the similar function of “translating” one representation of the same information to another one. This suggests that “information” itself should be defined in a translation invariant way. This “translation-invariant” quality is precisely how we seek to characterize information.
To introduce our formal definition for information elements to capture the essence of information itself, we note that information theory is built within the probabilistic framework, in which one-time information sources are usually modeled by random variables. Therefore, we start in the following with the concept of informational equivalence between random variables and develop the formal concept of information elements from first principles.
Recall that, given a probability space (Ω, F , P) and a measurable space (S, S), a random variable is a measurable function from Ω to S. The set S is usually called the state space of the random variable, and S is a σ-algebra on S. The set Ω is usually called the sample space; F is a σ-algebra on Ω, usually called the event space; and P denotes a probability measure on the measurable space (Ω, F ).
To illustrate the idea of informational equivalence, consider a random variable X : Ω → S and another random variable X ′ = f (X), where the function f : S → S ′ is bijective. Certainly, the two random variables X and X ′ are technically different for they have different codomains.
However, it is intuitively clear that that they are “equivalent” in some sense. In particular, one can infer the exact state of X by observing that of X ′ , and vice versa. For this reason, we may say that the two random variables X and X ′ carry the same piece of information. Note that the σ-algebras induced by X and X ′ coincide with each other. In fact, two random variables such that the state of one can be inferred from that of the other induce the same σ-algebra. This leads to the following definition for information equivalence. Definition 1: We say that two random variables X and X ′ are informationally equivalent, denoted X ∼ = X ′ , if the σ-algebras induced by X and X ′ coincide.
It is easy to verify that the “being-informational-equivalent” relation is an equivalence relation.
The definition reflects our intuition, as demonstrate in the previous motivating examples, that two random variables carry the same piece information if and only if they induce the same σ-algebra.
This motivates the following definition for information elements to capture the syntactical essence of information itself. Definition 2: An information element is an equivalence class of random variables with respect to the “being-informationally-equivalent” relation.
We call the random variables in the equivalent class of an information element m representing random variables of m. Or,
This content is AI-processed based on open access ArXiv data.