Rationality Measurement and Theory for Reinforcement Learning Agents

This paper proposes a suite of rationality measures and associated theory for reinforcement learning agents, a property increasingly critical yet rarely explored. We define an action in deployment to be perfectly rational if it maximises the hidden true value function in the steepest direction. The expected value discrepancy of a policy’s actions against their rational counterparts, culminating over the trajectory in deployment, is defined to be expected rational risk; an empirical average version in training is also defined. Their difference, termed as rational risk gap, is decomposed into (1) an extrinsic component caused by environment shifts between training and deployment, and (2) an intrinsic one due to the algorithm’s generalisability in a dynamic environment. They are upper bounded by, respectively, (1) the $1$-Wasserstein distance between transition kernels and initial state distributions in training and deployment, and (2) the empirical Rademacher complexity of the value function class. Our theory suggests hypotheses on the benefits from regularisers (including layer normalisation, $\ell_2$ regularisation, and weight normalisation) and domain randomisation, as well as the harm from environment shifts. Experiments are in full agreement with these hypotheses. The code is available at https://github.com/EVIEHub/Rationality.

💡 Research Summary

The paper introduces a formal framework for quantifying the rationality of reinforcement‑learning (RL) agents, a property that becomes increasingly important as RL systems are deployed in safety‑critical domains such as robotics, autonomous driving, and finance. The authors define a perfectly rational action as one that maximizes the hidden true value function (Q^{*}_{\dagger}) in the current state. Based on this definition they construct a rational value loss for any action, and then aggregate it over the state distribution induced by a policy to obtain the expected rational value loss at each timestep. Summing across the episode horizon yields the expected rational value risk (R(\pi)). An empirical counterpart (\hat R(\pi)) is defined by averaging the same loss over the training trajectories.

The central object of study is the rational risk gap (\Delta R = R(\pi) - \hat R(\pi)), which measures how far the agent’s behavior in deployment deviates from the rational benchmark observed during training. Lemma 2 shows that (\Delta R) can be decomposed into two additive components:

1. Extrinsic rational gap – the discrepancy caused by a shift between the training environment ((p, p_0)) and the deployment environment ((p^{\dagger}, p^{\dagger}_0)). The authors bound this term by a linear combination of 1‑Wasserstein distances (W_1(p^{\dagger}, p)) and (W_1(p^{\dagger}_0, p_0)), scaled by Lipschitz constants of the state‑to‑value mapping ((L_s)) and the transition‑kernel‑to‑state‑distribution mapping ((L_p)), as well as the episode horizon (H).

2. Intrinsic rational gap – the gap arising from finite‑sample generalisation within the training environment. This term is bounded using the empirical Rademacher complexity (\hat{\mathcal R}h(Q{\Pi})) of the value‑function class (Q_{\Pi}). The bound also involves the Lipschitz constant of the policy‑to‑state‑distribution map ((L_{\Pi})), the cardinality of the action space (|A|), the number of training episodes (T), and a confidence parameter (\delta).

These bounds lead to three testable hypotheses: (i) regularisation techniques such as layer normalisation, (\ell_2) regularisation, and weight normalisation reduce the intrinsic gap by controlling hypothesis complexity; (ii) domain randomisation mitigates the extrinsic gap by shrinking the Wasserstein distance between training and deployment dynamics; (iii) any shift in environment dynamics harms rationality by inflating the extrinsic gap.

Empirical validation is performed with Deep Q‑Networks on the Taxi‑v3 and CliffWalking benchmarks. Experiments confirm that regularisation consistently lowers the empirical rational risk, domain randomisation improves robustness to transition changes, and artificially induced environment shifts increase the extrinsic component, thereby degrading overall rationality.

The paper’s contributions are threefold: (1) a novel suite of rationality metrics grounded in value‑function optimality; (2) a rigorous decomposition of rationality loss into extrinsic and intrinsic components with provable upper bounds; (3) a theoretical justification for common regularisation and domain‑randomisation practices in deep RL. Limitations include the reliance on an unobservable true value function, the practical difficulty of estimating Wasserstein distances and Lipschitz constants, and the focus on relatively simple discrete environments. Future work could extend the framework to continuous control, multi‑agent settings, and non‑stationary or adversarial deployment scenarios.