Sparse Reward Processes

We introduce a class of learning problems where the agent is presented with a series of tasks. Intuitively, if there is relation among those tasks, then the information gained during execution of one task has value for the execution of another task. Consequently, the agent is intrinsically motivated to explore its environment beyond the degree necessary to solve the current task it has at hand. We develop a decision theoretic setting that generalises standard reinforcement learning tasks and captures this intuition. More precisely, we consider a multi-stage stochastic game between a learning agent and an opponent. We posit that the setting is a good model for the problem of life-long learning in uncertain environments, where while resources must be spent learning about currently important tasks, there is also the need to allocate effort towards learning about aspects of the world which are not relevant at the moment. This is due to the fact that unpredictable future events may lead to a change of priorities for the decision maker. Thus, in some sense, the model “explains” the necessity of curiosity. Apart from introducing the general formalism, the paper provides algorithms. These are evaluated experimentally in some exemplary domains. In addition, performance bounds are proven for some cases of this problem.

💡 Research Summary

The paper introduces a novel learning framework called the Sparse Reward Process (SRP), designed to capture the challenges of lifelong learning where an agent faces a sequence of related tasks in an uncertain environment. Unlike conventional reinforcement‑learning (RL) settings that focus on a single objective with frequent feedback, SRP models a multi‑stage stochastic game between a learning agent and an opponent (or a natural stochastic environment). In each stage the agent selects an action in a state, receives a reward that is extremely sparse and whose occurrence time and location are unknown, and updates its belief about the underlying dynamics. Crucially, the transition and reward structure are shared across tasks, so information gathered while solving one task can be valuable for future tasks. This motivates an intrinsic drive to explore beyond the immediate needs of the current task—a formalization of curiosity.

The authors formalize SRP as follows. Let θ denote the unknown parameters governing state transitions and reward generation. At stage t the agent observes state s_t, chooses action a_t, and receives reward r_t drawn from a distribution conditioned on (s_t, a_t, θ). The opponent may adapt the distribution of future tasks, possibly adversarially, to model unpredictable changes in priorities. The agent’s objective is to maximize cumulative expected reward over an indefinite horizon while accounting for the value of information that reduces uncertainty about θ.

To operationalize this trade‑off, the paper proposes two algorithmic families. The first, called Information‑Reward Mixing (IRM), augments the standard expected‑reward criterion with an information‑gain term. Using a Bayesian posterior over θ, the expected reduction in entropy ΔH(θ|a) after taking action a is computed. The policy selects actions that maximize E