Distributional Active Inference

Optimal control of complex environments with robotic systems faces two complementary and intertwined challenges: efficient organization of sensory state information and far-sighted action planning. Because the reinforcement learning framework addresses only the latter, it tends to deliver sample-inefficient solutions. Active inference is the state-of-the-art process theory that explains how biological brains handle this dual problem. However, its applications to artificial intelligence have thus far been limited to extensions of existing model-based approaches. We present a formal abstraction of reinforcement learning algorithms that spans model-based, distributional, and model-free approaches. This abstraction seamlessly integrates active inference into the distributional reinforcement learning framework, making its performance advantages accessible without transition dynamics modeling.

💡 Research Summary

The paper tackles the dual challenge of organizing high‑dimensional sensory information and performing long‑horizon planning in robotic control. While reinforcement learning (RL) excels at planning, it often suffers from poor sample efficiency because it does not address the representation side. Active inference (AIF), a process theory from neuroscience, simultaneously optimizes perception and action by minimizing expected free energy, but its AI applications have remained confined to model‑based extensions and have not yielded state‑of‑the‑art performance.

The authors first re‑derive AIF from first principles of variational Bayesian inference and causal do‑calculus. They model the world as a joint distribution P_W(X,Y,S)=P_D(X|Y,S) P₀(Y,S) with Y representing actions and S latent perceptions. The ELBO for an observation x is decomposed into four terms: a decoder likelihood, a latent prior term, a reward term E