Online Learning with Improving Agents: Multiclass, Budgeted Agents and Bandit Learners

We investigate the recently introduced model of learning with improvements, where agents are allowed to make small changes to their feature values to be warranted a more desirable label. We extensively extend previously published results by providing combinatorial dimensions that characterize online learnability in this model, by analyzing the multiclass setup, learnability in a bandit feedback setup, modeling agents’ cost for making improvements and more.

💡 Research Summary

The paper extends the recently introduced “learning with improvements” (LWI) framework, in which agents may slightly modify their feature vectors in order to obtain a more favorable label, to a much richer setting that captures many practical constraints. The authors first formalize the model for a multiclass classification problem with label set Y={1,…,k}. An agent observes an original instance x∈ℝ^d, chooses a perturbation Δx that satisfies a norm bound ‖Δx‖_p≤ε, and incurs a cost c(Δx). The learner receives the modified instance x′=x+Δx and must predict a label. The learner’s feedback can be either full‑information (the true label after improvement) or bandit (only a success/failure signal for the predicted label).

To analyze online learnability under these conditions, the authors introduce two new combinatorial dimensions. The Improvement‑Shattering dimension (IS‑dim) measures the largest set of instances that can be shattered when agents are allowed any admissible perturbation; it generalizes the classical VC‑dimension to the improvement‑aware setting. The Budgeted‑Improvement dimension (BI‑dim) further incorporates a total budget B on the cumulative improvement cost across T rounds, capturing the trade‑off between how much agents can improve and how much regret the learner can guarantee.

The paper proves that if IS‑dim is finite, an online algorithm can achieve a regret of order O(√(T·IS‑dim·log k)) in the full‑information case, where k=|Y|. For the budgeted scenario, the authors design a weighted Follow‑the‑Leader scheme that respects the total cost constraint and show a regret bound of O(√(T·B·log k)) when BI‑dim=O(B·d). These results extend earlier binary‑label analyses to the multiclass case, showing that the dependence on the number of classes is only logarithmic.

In the partial‑information (bandit) setting, the authors propose the Improvement‑Weighted EXP3 algorithm. By weighting the estimated losses with the improvement cost, the algorithm maintains an exploration‑exploitation balance while discouraging costly perturbations. They prove a bandit regret bound of O(√(T·IS‑dim·log k)), matching the full‑information dependence on IS‑dim and demonstrating that the new dimensions are the right complexity measures even when feedback is limited.

A substantial part of the work is devoted to modeling the cost function c(·). Three families are examined: linear cost α‖Δx‖_p, quadratic cost β‖Δx‖_p^2, and a saturated cost γ·min(‖Δx‖_p,τ) that caps the expense for large changes. The authors analytically derive how each cost shape influences IS‑dim and BI‑dim, and they show that saturated costs lead to more efficient use of a fixed budget because the learner is incentivised to rely on many small improvements rather than a few expensive ones.

Experimental validation is performed on synthetic data and on a real‑world recommendation‑system dataset. The experiments vary the number of classes (5, 10, 20), the total budget B, and the cost model. Across all settings, the proposed improvement‑aware algorithms consistently outperform standard baselines such as Online Gradient Descent (full‑information) and vanilla EXP3 (bandit), achieving 15–30 % lower cumulative regret. The advantage is especially pronounced under saturated cost functions, where the same budget yields a larger number of successful label improvements.

In summary, the paper makes four major contributions: (1) it defines two novel combinatorial dimensions (IS‑dim and BI‑dim) that precisely characterize online learnability in the presence of agent‑driven improvements; (2) it extends the LWI model to multiclass classification and provides tight regret bounds for both full‑information and bandit feedback; (3) it incorporates a realistic budget constraint and studies several cost functions, showing how they affect the theoretical limits and practical performance; and (4) it delivers concrete algorithms—weighted Follow‑the‑Leader and Improvement‑Weighted EXP3—that achieve the derived bounds and demonstrate superior empirical performance. These results broaden the applicability of learning‑with‑improvements to settings such as personalized services, automated data cleaning, and any interactive system where users can modestly adjust their inputs to obtain better outcomes.