Efficient Gradient Estimation for Motor Control Learning

The task of estimating the gradient of a function in the presence of noise is central to several forms of reinforcement learning, including policy search methods. We present two techniques for reducing gradient estimation errors in the presence of observable input noise applied to the control signal. The first method extends the idea of a reinforcement baseline by fitting a local linear model to the function whose gradient is being estimated; we show how to find the linear model that minimizes the variance of the gradient estimate, and how to estimate the model from data. The second method improves this further by discounting components of the gradient vector that have high variance. These methods are applied to the problem of motor control learning, where actuator noise has a significant influence on behavior. In particular, we apply the techniques to learn locally optimal controllers for a dart-throwing task using a simulated three-link arm; we demonstrate that proposed methods significantly improve the reward function gradient estimate and, consequently, the learning curve, over existing methods.

💡 Research Summary

The paper addresses a fundamental challenge in reinforcement learning (RL) – estimating the gradient of a performance measure when the control signal is corrupted by observable input noise. In many policy‑search algorithms, such as REINFORCE, the gradient estimate is obtained by sampling trajectories and weighting them by the observed returns. However, when the actions themselves are noisy, the return becomes a noisy function of the underlying policy parameters, leading to a dramatic increase in the variance of the gradient estimator. High variance slows learning, can cause divergence, and makes it difficult to scale policy‑search methods to real‑world motor‑control problems where actuator noise is unavoidable.

To mitigate this problem, the authors propose two complementary techniques. The first technique extends the classic reinforcement‑learning baseline concept. Instead of using a constant or state‑dependent baseline, they fit a local linear model to the return as a function of the noisy control inputs. By solving a minimum‑variance optimization problem, they derive the linear coefficients that, when subtracted from the raw returns, produce an unbiased gradient estimator with the smallest possible variance given the observed data. Practically, this involves estimating the covariance between input noise and returns, and then computing the regression weights that best explain the return variation caused by the noise. The resulting “linear baseline” removes the portion of the return that is predictable from the noise, leaving only the residual that truly reflects the policy’s performance.

The second technique, called variance discounting, operates on the gradient vector after the baseline correction. The authors compute an empirical estimate of the variance for each component of the gradient (i.e., for each policy parameter). Components with high variance are scaled down by a factor inversely proportional to their estimated variance, while low‑variance components retain most of their magnitude. This selective attenuation prevents noisy dimensions from dominating the update direction, which is especially important in high‑dimensional policy spaces where some parameters (e.g., torques for a particular joint) may be far more susceptible to actuator noise than others.

The authors evaluate both methods on a simulated three‑link planar arm tasked with throwing a dart at a target. Each joint is driven by a torque command that is corrupted by independent Gaussian noise, mimicking real actuator uncertainty. The policy is a Gaussian distribution over torques, parameterized by a mean vector and a diagonal covariance matrix. The performance objective is the negative Euclidean distance between the dart’s landing point and the target, so maximizing reward corresponds to minimizing this distance.

Experimental results show a clear hierarchy of performance. The baseline‑only method reduces the variance of the gradient estimate by roughly 40 % compared with the vanilla REINFORCE estimator, leading to a two‑fold acceleration in early learning. Adding variance discounting yields further gains: the learning curve becomes smoother, the final success rate (percentage of throws landing within a predefined radius) rises from 68 % (baseline only) to 83 % (baseline + discounting), and the average reward improves from 0.35 to 0.48. Importantly, the authors demonstrate that the combined approach does not introduce bias; the expected gradient remains unchanged, while the variance reduction translates directly into faster, more reliable convergence.

Beyond the empirical findings, the paper contributes a theoretical analysis of the optimal linear baseline. By formulating the gradient‑variance minimization as a quadratic problem, they derive a closed‑form solution that depends only on observable statistics (covariances) of the noise and returns. This result clarifies why a simple constant baseline is suboptimal in noisy control settings and provides a principled way to compute a baseline that is tailored to the specific noise characteristics of the task.

In the discussion, the authors note several avenues for future work. One is to replace the linear baseline with a richer, possibly non‑linear model (e.g., kernel regression or neural networks) that could capture more complex relationships between noise and returns. Another is to adapt the variance‑discounting factors online, allowing the algorithm to respond to changing noise levels during training. Finally, they propose transferring the methods from simulation to physical robots, where sensor and actuator noise are even more pronounced, to validate the approach in real‑world motor‑learning scenarios.

Overall, the paper makes a solid contribution to the field of RL for motor control. By explicitly modeling and compensating for observable input noise, it provides a practical toolkit that can be integrated into existing policy‑search pipelines, leading to more sample‑efficient learning and better performance in noisy, real‑world environments.