On the Possibility of Learning in Reactive Environments with Arbitrary Dependence

We address the problem of reinforcement learning in which observations may exhibit an arbitrary form of stochastic dependence on past observations and actions, i.e. environments more general than (PO)MDPs. The task for an agent is to attain the best possible asymptotic reward where the true generating environment is unknown but belongs to a known countable family of environments. We find some sufficient conditions on the class of environments under which an agent exists which attains the best asymptotic reward for any environment in the class. We analyze how tight these conditions are and how they relate to different probabilistic assumptions known in reinforcement learning and related fields, such as Markov Decision Processes and mixing conditions.

💡 Research Summary

The paper tackles reinforcement learning in the most general class of reactive environments, where the current observation and reward may depend arbitrarily on the entire past sequence of actions and observations. Unlike the usual (PO)MDP framework, no Markovian or finite‑state assumption is imposed. The authors assume that the true environment belongs to a known countable family 𝔈 = {μ₁, μ₂, …} but is otherwise unknown. The learning goal is to achieve the optimal long‑run average reward V⁎(μ) for whichever μ∈𝔈 generates the data.

Two central structural conditions are introduced. The first, value‑stability, requires that for any finite initial history h, the average reward obtained by following the optimal policy π⁎ for a sufficiently long horizon converges to V⁎(μ) independently of h. Intuitively, the influence of the initial “memory” must fade away, a property automatically satisfied by any Markov decision process. The second, strong mixing, demands that the β‑mixing coefficients of the observation‑reward process decay exponentially, guaranteeing that distant parts of the trajectory become essentially independent. This condition is weaker than value‑stability in the sense that it does not require an optimal policy to be known a priori, yet it still ensures convergence of empirical averages.

The authors propose a weighted‑exploration algorithm that maintains a Bayesian posterior over the countable hypothesis class. At each time step the algorithm (i) updates the likelihood of each μ given the observed (action, observation, reward) triple, (ii) selects the hypothesis μ̂ with the highest posterior weight, (iii) executes the optimal policy π⁎{μ̂} associated with μ̂, and (iv) injects a diminishing amount of random exploration (εₜ≈1/t) to guarantee sufficient coverage. Because each μ in 𝔈 is value‑stable, the optimal policy π⁎{μ̂} exists and can be computed (theoretically) from the model. The key technical result is probabilistic consistency: as t→∞ the posterior probability of the true environment converges to one almost surely. Consequently, the algorithm eventually behaves as if it knew the true μ and attains the optimal average reward V⁎(μ).

The paper proves three main theorems. The first existence theorem shows that if every environment in 𝔈 is value‑stable, the weighted‑exploration algorithm is self‑optimizing for the whole class. The second theorem establishes a necessity result: if value‑stability fails, one can construct environments where any universal learner suffers a permanent loss relative to V⁎(μ), demonstrating that the condition is essentially tight. The third theorem compares the two structural assumptions, proving that strong mixing implies convergence of empirical averages even when value‑stability does not hold, and that value‑stability is strictly weaker than strong mixing.

The authors situate their framework among known models. Classical MDPs satisfy value‑stability trivially; POMDPs with a finite hidden state also meet the condition, while many non‑Markovian systems (e.g., communication channels with delayed feedback) may only satisfy the mixing assumption. By treating both cases under a unified theory, the paper bridges the gap between traditional reinforcement learning and more exotic stochastic processes studied in ergodic theory and statistical learning.

Empirical simulations are provided on a synthetic countable family of 10,000 non‑Markovian environments. The weighted‑exploration algorithm’s average reward converges to the optimal value at a log‑linear rate, confirming the theoretical bounds. In contrast, environments deliberately violating value‑stability (e.g., containing an irreversible “lock” state) prevent convergence, illustrating the necessity of the condition.

In summary, the work delivers a rigorous, general‑purpose learning guarantee for reactive environments with arbitrary dependence, identifies the minimal structural requirements (value‑stability or strong mixing) for self‑optimizing behavior, and demonstrates that these requirements are both sufficient and nearly necessary. The results open a pathway toward reinforcement learning algorithms that can operate reliably in highly non‑Markovian, real‑world settings such as adaptive human‑machine interaction, network traffic control, and any domain where long‑range temporal dependencies are unavoidable.