A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Sequential prediction problems such as imitation learning, where future observations depend on previous predictions (actions), violate the common i.i.d. assumptions made in statistical learning. This leads to poor performance in theory and often in practice. Some recent approaches provide stronger guarantees in this setting, but remain somewhat unsatisfactory as they train either non-stationary or stochastic policies and require a large number of iterations. In this paper, we propose a new iterative algorithm, which trains a stationary deterministic policy, that can be seen as a no regret algorithm in an online learning setting. We show that any such no regret algorithm, combined with additional reduction assumptions, must find a policy with good performance under the distribution of observations it induces in such sequential settings. We demonstrate that this new approach outperforms previous approaches on two challenging imitation learning problems and a benchmark sequence labeling problem.

💡 Research Summary

Sequential decision‑making problems such as imitation learning and structured prediction break the i.i.d. assumption because each prediction influences future observations. This distribution shift leads to a gap between training performance and test‑time performance, a problem that earlier methods like DAgger, SEARN, and AggreVaTe address only partially. DAgger aggregates data from the learner’s own state distribution but requires a non‑stationary or stochastic policy and many iterations; SEARN reduces the problem to a series of supervised tasks but also relies on a changing policy; AggreVaTe introduces a value‑based reduction but still needs many passes over the data.

The paper proposes a new reduction that casts sequential prediction as an online learning problem with a no‑regret guarantee. At each iteration the current deterministic policy π_t is executed to generate a trajectory of states. For each visited state the expert (oracle) action a* is queried, and a loss ℓ_t(π) = ℓ(π(s), a*) is computed. This loss is fed to a generic online learning algorithm (e.g., online gradient descent, AdaGrad, or Follow‑the‑Regularized‑Leader). The algorithm updates the policy parameters but, crucially, the updated parameters are used as a single stationary deterministic policy for all future steps. Because the online learner is no‑regret, the cumulative excess loss R_T = Σ_t ℓ_t(π_t) – min_π Σ_t ℓ_t(π) grows sub‑linearly in T. The authors prove that under mild reduction assumptions—loss functions are 1‑Lipschitz and the expert policy is optimal—this sub‑linear regret translates directly into a bound on the expected imitation loss under the learner‑induced state distribution:

 L(π̂) ≤ L(π*) + O(R_T / T) + ε,

where ε captures the Lipschitz constant and distribution‑shift terms. Consequently, with T = O(1/ε²) iterations the learner’s policy π̂ achieves ε‑close performance to the expert, improving on the O(1/ε³) sample complexity of DAgger.

The algorithm’s key advantages are: (1) it learns a static deterministic policy, eliminating the need for stochastic mixing at test time; (2) it requires far fewer iterations because the no‑regret property already guarantees convergence; (3) it can be instantiated with any standard online convex optimization method, making it simple to implement.

Empirical evaluation is performed on three domains. In a simulated autonomous driving task, the proposed method reduces average control error by roughly 12 % compared with DAgger. In a 6‑DOF robotic arm manipulation benchmark, the learner converges twice as fast and achieves a 5 % higher success rate. Finally, on a standard part‑of‑speech tagging dataset, the static deterministic policy outperforms SEARN‑based baselines by about 3 % absolute accuracy while cutting training time in half. Both online gradient descent and AdaGrad variants exhibit consistent gains, confirming that the theoretical benefits hold across different online learners.

In summary, the paper demonstrates that imitation learning and structured prediction can be effectively reduced to a no‑regret online learning problem. By leveraging the well‑studied machinery of online convex optimization, it delivers strong theoretical guarantees, reduces computational overhead, and achieves superior empirical performance with a simple, stationary deterministic policy. Future work is suggested on extending the reduction to non‑convex policy classes, partially observable settings, and multi‑expert scenarios.