How to Purchase Labels? A Cost-Effective Approach Using Active Learning Markets

We introduce and analyse active learning markets as a way to purchase labels, in situations where analysts aim to acquire additional data to improve model fitting, or to better train models for predictive analytics applications. This comes in contrast to the many proposals that already exist to purchase features and examples. By originally formalising the market clearing as an optimisation problem, we integrate budget constraints and improvement thresholds into the label acquisition process. We focus on a single-buyer-multiple-seller setup and propose the use of two active learning strategies (variance based and query-by-committee based), paired with distinct pricing mechanisms. They are compared to benchmark baselines including random sampling and a greedy knapsack heuristic. The proposed strategies are validated on real-world datasets from two critical application domains: real estate pricing and energy forecasting. Results demonstrate the robustness of our approach, consistently achieving superior performance with fewer labels acquired compared to conventional methods. Our proposal comprises an easy-to-implement practical solution for optimising data acquisition in resource-constrained environments.

💡 Research Summary

The paper introduces “active learning markets” as a novel framework for purchasing labels under budget constraints and performance improvement requirements. Unlike existing data markets that focus on buying whole observations or additional features, this work targets the acquisition of missing target values (labels) while the buyer already possesses the full feature set. The market is modeled as a single‑buyer‑multiple‑seller scenario: the buyer holds a labeled set D_L and an unlabeled feature set D_U, while each seller S_j owns a single label y_j. The buyer’s willingness to pay (φ), total budget (B), and a target performance reduction (α) are defined a priori.

The label acquisition problem is formalized as an integer optimization that maximizes expected performance gain subject to the budget, effectively a knapsack‑type problem where each potential label has both a monetary cost and an informational value. To estimate the informational value, the authors embed two classic active‑learning strategies. The first, variance‑based active learning (VBAL), selects points with the highest predictive variance under a linear regression model. The second, query‑by‑committee active learning (QBCAL), selects points where a committee of models disagrees most. Each strategy is paired with two pricing mechanisms: a fixed unit price derived from φ and a variable price that reflects each seller’s reported cost.

Experiments are conducted on two real‑world domains: (1) UK residential property pricing, where many house features are known but sale prices are scarce, and (2) energy consumption forecasting for an educational building, where sensor features are abundant but actual consumption labels are limited. Starting from an initial labeled pool K, the authors compare four combinations (VBAL‑Fixed, VBAL‑Variable, QBCAL‑Fixed, QBCAL‑Variable) against two baselines: random sampling and a greedy knapsack heuristic. Performance is measured by mean‑squared error (MSE) reduction per acquired label. Results show that all active‑learning‑market approaches achieve lower MSE with fewer labels than the baselines; QBCAL‑Variable consistently yields the highest label‑efficiency, reducing required labels by roughly 20‑30 % for the same performance target. The variable pricing scheme also improves budget utilization compared with the fixed price.

The study’s contributions are threefold: (i) a formal market‑clearing formulation for label purchase that integrates budget and performance thresholds, (ii) a practical algorithmic coupling of active‑learning selection with simple pricing rules, and (iii) empirical validation on two critical industry datasets demonstrating cost‑effective label acquisition. Limitations include the reliance on linear regression (restricting applicability to more complex models), the assumption of one label per seller, a static willingness‑to‑pay, and a batch‑only acquisition setting. The authors suggest future work on extending to non‑linear models, multi‑label sellers, multi‑buyer competition, and adaptive pricing mechanisms in online environments. Overall, the paper provides a compelling blueprint for organizations seeking to strategically purchase high‑value labels while respecting tight financial constraints.