Hypercube Policy Regularization Framework for Offline Reinforcement Learning

Offline reinforcement learning has received extensive attention from scholars because it avoids the interaction between the agent and the environment by learning a policy through a static dataset. However, general reinforcement learning methods cannot get satisfactory results in offline reinforcement learning due to the out-of-distribution state actions that the dataset cannot cover during training. To solve this problem, the policy regularization method that tries to directly clone policies used in static datasets has received numerous studies due to its simplicity and effectiveness. However, policy constraint methods make the agent choose the corresponding actions in the static dataset. This type of constraint is usually over-conservative, which results in suboptimal policies, especially in low-quality static datasets. In this paper, a hypercube policy regularization framework is proposed, this method alleviates the constraints of policy constraint methods by allowing the agent to explore the actions corresponding to similar states in the static dataset, which increases the effectiveness of algorithms in low-quality datasets. It was also theoretically demonstrated that the hypercube policy regularization framework can effectively improve the performance of original algorithms. In addition, the hypercube policy regularization framework is combined with TD3-BC and Diffusion-QL for experiments on D4RL datasets which are called TD3-BC-C and Diffusion-QL-C. The experimental results of the score demonstrate that TD3-BC-C and Diffusion-QL-C perform better than state-of-the-art algorithms like IQL, CQL, TD3-BC and Diffusion-QL in most D4RL environments in approximate time.

💡 Research Summary

Offline reinforcement learning (Offline RL) seeks to learn policies from a fixed dataset without interacting with the environment, which eliminates safety risks and reduces training costs. However, static datasets rarely cover the entire state‑action space, leading to distribution‑shift problems: Q‑functions tend to over‑estimate out‑of‑distribution (OOD) state‑action pairs, causing the learned policy to select actions that appear high‑valued but are unsupported by data. Existing approaches to mitigate this issue fall into several categories, among which policy regularization and Q‑value regularization are the most prominent.

Policy regularization methods (e.g., BCQ, BEAR, TD3‑BC, IQL, Diffusion‑QL) constrain the learned policy to stay close to the behavior policy that generated the dataset. This yields stable training but becomes overly conservative when the dataset is of low quality, because the agent is forced to imitate sub‑optimal actions. Q‑value regularization methods (e.g., CQL, EDA‑C, PBRL) penalize Q‑values of OOD actions, encouraging exploration, yet they require tight Q‑value bounds and incur substantial computational overhead. The trade‑off is clear: stability versus exploration.

The paper introduces a “Hypercube Policy Regularization Framework” that aims to combine the strengths of both families while avoiding their drawbacks. The central idea is to partition the state space into hypercubes using an integer granularity parameter δ. Each state s is first normalized to a coordinate s′ via a modular scaling (Eq. 1) and then mapped to an integer index v(s′) (Eq. 2). All state‑action pairs that share the same index belong to the same hypercube, effectively clustering “similar” states.

During training, for a given state s the algorithm looks up all actions a′ associated with other states in the same hypercube. It evaluates the Q‑value Q(s, a′) for each of these actions and selects the action a_max with the highest Q‑value (Eq. 3). If Q(s, a_max) ≥ Q(s, a) (where a is the action suggested by the current policy), the algorithm replaces a with a_max when computing the policy‑regularization loss. In this way, the agent is allowed to explore actions that have been observed in nearby states, rather than being forced to repeat the exact action from the exact state in the dataset. This “local exploration” preserves the stability of policy regularization while providing a controlled avenue for improvement, especially when the dataset contains noisy or sub‑optimal actions.

Theoretical analysis assumes the Q‑function is Lipschitz‑continuous (Assumption 1). Under this assumption, Theorem 1 proves that if the hypercube granularity δ (or equivalently the derived parameter θ) is chosen sufficiently large, the new policy π_new will never have a lower Q‑value than the original policy π_old for any state: Q(s, π_new(s)) ≥ Q(s, π_old(s)). The proof examines three cases—Q(s, a) < Q(s′, a′), equality, and Q(s, a) > Q(s′, a′)—and shows that the distance between states within a hypercube is bounded by S_max, which can be made small by increasing δ. Consequently, the Q‑value difference dominates the Lipschitz term, guaranteeing non‑degradation. The analysis also extends to multiple actions per hypercube, concluding that a sufficiently large δ ensures monotonic improvement or at least no loss. Practically, because Q‑function approximators are reasonably accurate on in‑dataset points, a smaller δ can be used to increase exploration, while a larger δ yields a more conservative update.

To demonstrate practicality, the framework is integrated with two state‑of‑the‑art offline RL algorithms:

1. TD3‑BC‑C – The authors augment the TD3‑BC algorithm with a hypercube module. For each sampled transition, they compute a_max from the hypercube, compare Q(s, a) and Q(s, a_max), and replace a with a_max when updating the policy regularization loss (Eq. 14). The rest of TD3‑BC (actor‑critic updates, target networks) remains unchanged.

2. Diffusion‑QL‑C – Diffusion‑QL already uses a conditional diffusion model to clone the behavior policy. The authors replace the sampled actions from the diffusion model with a_max drawn from the hypercube, thereby injecting locally optimal actions while still benefiting from the expressive diffusion prior.

Both variants retain the original training time because the hypercube lookup and a_max computation are O(1) operations after an initial preprocessing pass that builds the hypercube index.

Empirical evaluation is conducted on the D4RL benchmark, covering a range of tasks (e.g., half‑cheetah, hopper, walker2d, ant‑maze) with varying dataset qualities (expert, medium, medium‑replay, medium‑expert). Results show that TD3‑BC‑C and Diffusion‑QL‑C consistently outperform their base counterparts and several strong baselines (IQL, CQL, standard TD3‑BC, original Diffusion‑QL). The most pronounced gains appear in low‑quality datasets such as medium‑replay and medium‑expert, where the proposed local exploration yields 20‑30 % higher normalized scores. In high‑quality expert datasets the improvements are modest but still non‑negative, confirming the theoretical guarantee of non‑degradation.

In summary, the paper makes three key contributions:

• A novel hypercube‑based state clustering mechanism that enables controlled local exploration without sacrificing the stability of policy regularization.
• Theoretical guarantees that, under mild Lipschitz assumptions, the framework cannot hurt performance and can improve it when the hypercube granularity is appropriately chosen.
• Practical algorithms (TD3‑BC‑C, Diffusion‑QL‑C) that integrate seamlessly with existing offline RL pipelines, retain comparable computational cost, and achieve state‑of‑the‑art results on a wide suite of benchmarks, especially in settings where the dataset is noisy or sub‑optimal.

The work therefore offers a compelling middle ground between overly conservative behavior cloning and computationally heavy Q‑value regularization, opening a new direction for robust and efficient offline reinforcement learning.