Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies

This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation. The algorithm is based on two innovations. Firstly, we present a temporal-difference based method for learning the gradient of the value-function. Secondly, we present the deviator-actor-critic (DAC) model, which comprises three neural networks that estimate the value function, its gradient, and determine the actor’s policy respectively. We evaluate GProp on two challenging tasks: a contextual bandit problem constructed from nonparametric regression datasets that is designed to probe the ability of reinforcement learning algorithms to accurately estimate gradients; and the octopus arm, a challenging reinforcement learning benchmark. GProp is competitive with fully supervised methods on the bandit task and achieves the best performance to date on the octopus arm.

💡 Research Summary

The paper introduces GProp, a deep reinforcement‑learning algorithm designed for continuous‑action domains that satisfies the compatible function‑approximation conditions required for unbiased deterministic policy‑gradient updates. The authors address two longstanding issues: (1) traditional temporal‑difference (TD) methods estimate the value function but not its gradient, and (2) existing compatible approximations tie the critic’s representation tightly to the actor’s parameters, making back‑propagation infeasible for nonlinear policies.

To solve (1), they develop a “gradient perturbation trick” (Lemma 3). By adding isotropic Gaussian noise of variance σ² to the input of an unknown scalar function f and minimizing the mean‑squared error between the noisy output and a linear model, the optimal weight vector converges to ∇f at the perturbation point. This technique integrates naturally with TD learning, allowing simultaneous estimation of the value function Q(s,a) and its action‑gradient G(s,a).

For (2) they propose the Deviator‑Actor‑Critic (DAC) architecture, comprising three coupled neural networks: a Critic that learns V(s), a Deviator that learns G(s,a)≈∇ₐQ(s,a), and an Actor that outputs a deterministic policy μθ(s). Each network has its own parameter set (Θ, W, V) and receives gradients only from the others’ outputs, so the actor’s gradient update is simply ∇θμθ(s)·G(s,μθ(s)). Theorem 6 proves that, when the networks consist of linear or ReLU units, DAC satisfies both compatibility conditions: C1 (the critic approximates the true value gradient) and C2 (the gradient of the value estimate aligns with the policy Jacobian).

Empirically, GProp is evaluated on two challenging benchmarks. First, the authors transform the SARCOS and Barrett robotic regression datasets into continuous contextual bandit problems where the reward is the negative Euclidean distance to the true torque vector. Accurate gradient estimation is essential; GProp achieves performance comparable to fully supervised regression models despite never observing the true labels. Second, on the Octopus Arm task—a high‑dimensional, long‑horizon control problem—GProp outperforms all previously reported methods, establishing a new state‑of‑the‑art result.

The paper situates GProp relative to prior work: REINFORCE, Deep Q‑Learning, deterministic policy gradient (Silver et al., 2014), and COPDAC‑Q. Unlike these, GProp directly learns the value gradient, decouples actor and critic representations, and provides a rigorous compatibility proof for nonlinear policies. The authors also discuss connections to multi‑agent credit assignment and structural decomposition of neural units.

In summary, GProp contributes (i) a novel TD‑compatible gradient‑estimation technique, (ii) the DAC three‑network architecture enabling unbiased policy‑gradient updates for deep, continuous policies, (iii) theoretical guarantees of compatibility for networks with linear/ReLU units, and (iv) strong empirical results on both synthetic bandit settings and a demanding robotic control benchmark. This work advances the practical applicability of deep actor‑critic methods in continuous domains.