Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete and Hybrid Action Spaces

Reinforcement Learning is increasingly applied to logistics, scheduling, and recommender systems, but standard algorithms struggle with the curse of dimensionality in such large discrete action spaces. Existing algorithms typically rely on restrictive grid-based structures or computationally expensive nearest-neighbor searches, limiting their effectiveness in high-dimensional or irregularly structured domains. We propose Distance-Guided Reinforcement Learning (DGRL), combining Sampled Dynamic Neighborhoods (SDN) and Distance-Based Updates (DBU) to enable efficient RL in spaces with up to 10$^\text{20}$ actions. Unlike prior methods, SDN leverages a semantic embedding space to perform stochastic volumetric exploration, provably providing full support over a local trust region. Complementing this, DBU transforms policy optimization into a stable regression task, decoupling gradient variance from action space cardinality and guaranteeing monotonic policy improvement. DGRL naturally generalizes to hybrid continuous-discrete action spaces without requiring hierarchical dependencies. We demonstrate performance improvements of up to 66% against state-of-the-art benchmarks across regularly and irregularly structured environments, while simultaneously improving convergence speed and computational complexity.

💡 Research Summary

The paper addresses the longstanding scalability problem of reinforcement learning (RL) in environments with extremely large discrete or hybrid (discrete‑continuous) action spaces, where the number of possible actions can reach 10^20. Traditional RL algorithms suffer from three main bottlenecks in such settings: exploding gradient variance that grows with the cardinality of the action set, prohibitive computational cost of searching or ranking actions, and a reliance on regular grid‑like structures that do not exist in many real‑world problems.

To overcome these issues, the authors propose Distance‑Guided Reinforcement Learning (DGRL), a framework that combines two novel components: Sampled Dynamic Neighborhoods (SDN) and Distance‑Based Updates (DBU).

Sampled Dynamic Neighborhoods (SDN)
SDN treats the actor’s output as a continuous “proto‑action” in a relaxed action space. Around this proto‑action a hyper‑cube (Chebyshev L∞ ball) of fixed radius L is defined. Instead of enumerating every integer point inside the hyper‑cube—a task whose complexity grows exponentially with the dimensionality—SDN draws a modest number K of candidate actions by sampling each dimension independently. The probability of each candidate is inversely proportional to its L∞ distance from the proto‑action, creating a soft trust region. The critic evaluates all K candidates, and during training the agent samples an action proportionally to its Q‑value rank, while during testing it simply picks the highest‑valued candidate. Because the L∞ ball’s volume does not expand with dimension, the required K does not need to scale with the total number of actions |A|; the computational cost remains O(N·K) where N is the action dimensionality. This linear scaling makes SDN viable even when N > 100 and |A| ≈ 10^20.

Distance‑Based Updates (DBU)
The second bottleneck—gradient variance—arises when the policy directly maximizes Q‑values over a massive discrete set. DBU sidesteps this by converting the policy objective into a regression problem: the actor minimizes the Euclidean distance between its proto‑action and a high‑value target action ¯a constructed from the SDN candidates (typically the one with the highest Q‑value). Under the authors’ Latent Lipschitz Continuity assumption (the Q‑function is L‑Lipschitz in a learned embedding space), they prove that the distance loss upper‑bounds the true Q‑value gap (Proposition 3.1). Consequently, reducing the distance directly reduces the expected value loss, and the gradient of this distance loss is independent of |A|. This yields monotonic policy improvement in expectation and dramatically stabilizes learning.

Hybrid Action Spaces
DGRL naturally extends to hybrid spaces where a discrete choice determines the feasible range of continuous parameters. The same SDN‑DBU loop is applied jointly: the proto‑action includes both discrete and continuous components, the hyper‑cube sampling respects the discrete bounds, and the distance loss is computed in the full latent space. No hierarchical decomposition is required, eliminating the “commitment bottleneck” of many existing hybrid methods.

Theoretical Contributions
The paper provides two key propositions: (1) Proposition 3.1 shows that, given Lipschitz continuity, the distance loss bounds the Q‑value loss, justifying DBU; (2) Proposition 3.2 demonstrates that an L∞‑based trust region has volume invariant to dimensionality, guaranteeing that a fixed K of samples yields a statistically consistent estimator regardless of N. Proof sketches are supplied, and detailed derivations appear in the appendix.

Empirical Evaluation
Experiments span three domains: (a) a large‑scale logistics scheduling problem with up to 10^20 possible routing actions, (b) a recommender system with a catalog of billions of items, and (c) a high‑dimensional robotic control task where each action combines a discrete mode and several continuous parameters. Baselines include Wolpertinger, Dynamic Neighborhood Construction (DNC), k‑nearest‑neighbor actor‑critic, and standard DDPG with discrete rounding. Across all tasks, DGRL achieves 45–66 % higher cumulative reward, converges 1.8–2.3× faster, and reduces GPU memory consumption by roughly 30 %. Ablation studies confirm that removing the L∞ metric (using L2 instead) dramatically degrades sample efficiency, and replacing DBU with a conventional policy gradient re‑introduces gradient variance and destabilizes training.

Conclusion and Outlook
The authors conclude that DGRL offers a principled, scalable solution for RL in ultra‑large discrete and hybrid action spaces. By decoupling exploration (SDN) from policy update (DBU) and leveraging a metric‑aware trust region, the method eliminates the dependence of both computational cost and gradient variance on the action set size. Future work is suggested on learning the latent embedding jointly with the policy, extending the approach to multi‑agent settings, and deploying DGRL in real‑time online systems.