Progress Constraints for Reinforcement Learning in Behavior Trees

Behavior Trees (BTs) provide a structured and reactive framework for decision-making, commonly used to switch between sub-controllers based on environmental conditions. Reinforcement Learning (RL), on the other hand, can learn near-optimal controllers but sometimes struggles with sparse rewards, safe exploration, and long-horizon credit assignment. Combining BTs with RL has the potential for mutual benefit: a BT design encodes structured domain knowledge that can simplify RL training, while RL enables automatic learning of the controllers within BTs. However, naive integration of BTs and RL can lead to some controllers counteracting other controllers, possibly undoing previously achieved subgoals, thereby degrading the overall performance. To address this, we propose progress constraints, a novel mechanism where feasibility estimators constrain the allowed action set based on theoretical BT convergence results. Empirical evaluations in a 2D proof-of-concept and a high-fidelity warehouse environment demonstrate improved performance, sample efficiency, and constraint satisfaction, compared to prior methods of BT-RL integration.

💡 Research Summary

The paper addresses a critical shortcoming in the integration of Behavior Trees (BTs) with Reinforcement Learning (RL): the tendency of learned sub‑controllers to undo progress made by other parts of the tree, leading to oscillations, unsafe actions, or unnecessary repetitions. The authors propose “progress constraints,” a mechanism that leverages BT convergence theory to identify invariant sets—called convergence sets—associated with each leaf node. By ensuring that each RL controller respects its convergence set, the overall BT is guaranteed to move monotonically toward the global success region.

The technical foundation starts with a formal definition of BT nodes, their return statuses (Running, Success, Failure), and the partition of the state space into influence regions (I_i) and operating regions (Ω_i). Building on recent convergence results (reference