On the Information Rates of the Plenoptic Function

The {\it plenoptic function} (Adelson and Bergen, 91) describes the visual information available to an observer at any point in space and time. Samples of the plenoptic function (POF) are seen in video and in general visual content, and represent large amounts of information. In this paper we propose a stochastic model to study the compression limits of the plenoptic function. In the proposed framework, we isolate the two fundamental sources of information in the POF: the one representing the camera motion and the other representing the information complexity of the “reality” being acquired and transmitted. The sources of information are combined, generating a stochastic process that we study in detail. We first propose a model for ensembles of realities that do not change over time. The proposed model is simple in that it enables us to derive precise coding bounds in the information-theoretic sense that are sharp in a number of cases of practical interest. For this simple case of static realities and camera motion, our results indicate that coding practice is in accordance with optimal coding from an information-theoretic standpoint. The model is further extended to account for visual realities that change over time. We derive bounds on the lossless and lossy information rates for this dynamic reality model, stating conditions under which the bounds are tight. Examples with synthetic sources suggest that in the presence of scene dynamics, simple hybrid coding using motion/displacement estimation with DPCM performs considerably suboptimally relative to the true rate-distortion bound.

💡 Research Summary

The plenoptic function (POF) captures the complete visual information that an observer can obtain at any point in space and time. Because modern visual media—video, light‑field capture, VR/AR—sample this function, understanding its fundamental information limits is essential for designing efficient compression schemes. In this paper the authors introduce a stochastic framework that isolates the two primary sources of information in the POF: (1) the camera motion (the observer’s trajectory through space‑time) and (2) the intrinsic complexity of the “reality” being observed (the statistical structure of the scene). By modeling each source as an independent random process and then combining them, they obtain a composite stochastic process that faithfully represents sampled POF data.

Static‑Reality Model
The first part of the analysis assumes a scene that does not change over time. The scene is modeled as a random texture field (e.g., a stationary Markov random field) with a well‑defined entropy rate. The camera trajectory is treated either as known a priori or as an unknown sequence generated by an i.i.d. or Markov process. When the trajectory is known, the total entropy of the sampled POF reduces to the entropy of the texture field alone; optimal lossless coding therefore needs to compress only the scene. When the trajectory is unknown, an additional overhead is required to encode the motion parameters. The authors derive exact lossless rate bounds for both cases and show that the overhead matches the bit‑cost of motion‑vector signalling in current video codecs. For lossy compression, they consider a Gaussian texture model and obtain a closed‑form rate‑distortion (R‑D) function that coincides with the classical Shannon R‑D curve. Empirical tests with HEVC‑style transform coding confirm that modern video codecs operate very close to this theoretical optimum for static scenes.

Dynamic‑Reality Model
The second part extends the framework to scenes that evolve over time. The scene dynamics are modeled as an independent stochastic process (e.g., AR(1), Markov‑switching) while the camera motion remains independent. The total lossless rate is bounded above by the sum of the entropy rates of the motion process and the scene‑change process. For lossy compression, the authors derive R‑D bounds for Gaussian dynamic textures, showing that the distortion is governed by the power‑spectral density of the temporal variations. They compare these bounds with a hybrid coding scheme that combines motion‑compensated prediction and differential pulse‑code modulation (DPCM). Synthetic experiments reveal that, as the temporal variation of the scene becomes faster or its spatial complexity higher, the hybrid scheme deviates dramatically from the optimal R‑D curve, consuming many times more bits for the same distortion. This demonstrates that simple motion‑estimation plus DPCM is fundamentally sub‑optimal for dynamic plenoptic data.

Complexity Quantification
A notable contribution is the quantitative link between Kolmogorov complexity and entropy rate, which the authors use to define “scene information complexity.” Low‑complexity scenes (e.g., static backgrounds) have low entropy rates and can be transmitted with very few bits, whereas high‑complexity scenes (rich textures, rapid motion) demand substantially higher rates. This insight provides a principled way to allocate bandwidth in light‑field and immersive‑media pipelines.

Experimental Validation
Using synthetic Gaussian textures and controlled camera paths, the authors validate the theoretical bounds. For static scenes, HEVC‑style transform coding nearly attains the lossless and lossy limits. For dynamic scenes, even state‑of‑the‑art codecs (HEVC, AV1) fall short of the derived R‑D bounds, confirming the need for more sophisticated predictive models that explicitly capture temporal scene statistics.

Implications and Future Directions
The work shows that current video compression is essentially optimal for static plenoptic data but leaves a large gap for dynamic data. Closing this gap will likely require (i) explicit modeling of scene dynamics within the codec (e.g., learned temporal predictors, multi‑scale transforms), and (ii) more efficient representation of camera trajectories (e.g., trajectory priors, compressed sensing of motion). The stochastic model presented offers a rigorous foundation for such developments and can be extended to multi‑view, light‑field, and VR/AR systems where the full plenoptic function is sampled. In summary, the paper provides precise information‑theoretic limits for both lossless and lossy coding of the plenoptic function, demonstrates where existing hybrid motion‑compensation/DPCM approaches fall short, and points toward a new generation of codecs that integrate motion and scene‑complexity models to approach the true rate‑distortion frontier.