An Active Learning Approach for Jointly Estimating Worker Performance and Annotation Reliability with Crowdsourced Data

Crowdsourcing platforms offer a practical solution to the problem of affordably annotating large datasets for training supervised classifiers. Unfortunately, poor worker performance frequently threatens to compromise annotation reliability, and requesting multiple labels for every instance can lead to large cost increases without guaranteeing good results. Minimizing the required training samples using an active learning selection procedure reduces the labeling requirement but can jeopardize classifier training by focusing on erroneous annotations. This paper presents an active learning approach in which worker performance, task difficulty, and annotation reliability are jointly estimated and used to compute the risk function guiding the sample selection procedure. We demonstrate that the proposed approach, which employs active learning with Bayesian networks, significantly improves training accuracy and correctly ranks the expertise of unknown labelers in the presence of annotation noise.

💡 Research Summary

The paper tackles two fundamental challenges in crowdsourced data annotation: the high cost of acquiring large numbers of labels and the degradation of label quality caused by variable worker performance and task difficulty. While traditional active learning methods reduce the number of required samples, they are vulnerable to label noise because a single erroneous annotation can misguide the learning process. Conversely, the GLAD (Generative model of Labels, Abilities, and Difficulties) framework jointly estimates worker expertise (α), task difficulty (β), and true labels (z) using a Bayesian network and Expectation‑Maximization (EM), but it does not incorporate any mechanism to limit labeling cost.

The authors propose a novel integration of GLAD with an active‑learning loop that selects the most informative “worker‑task” pair at each iteration. The risk function guiding selection is the entropy of the posterior label distribution for each task:
Risk(z_j) = – Σ_{c∈C} p(z_j=c | L, θ) log p(z_j=c | L, θ).
Tasks with the highest entropy are the most uncertain, and the algorithm queries the worker currently estimated to have the highest expertise (largest α̂) for that task. This “uncertain task + reliable worker” strategy directly targets the sources of noise while keeping the number of queries low.

The learning cycle proceeds as follows: (1) run EM on the current partially labeled matrix to update α, β, and the posterior over true labels; (2) compute the entropy‑based risk for every unlabeled task; (3) pick the task with maximal risk; (4) request a label from the top‑ranked worker who has not yet labeled that task; (5) decrement the labeling budget and repeat until the budget is exhausted. The authors also explore alternative worker‑selection policies: weighted sampling proportional to α̂, and an ε‑greedy scheme that occasionally selects a non‑top worker to encourage exploration.

Experiments are conducted on two fronts. First, a synthetic benchmark replicates the setting of Whitehill et al., with 100 workers and 5,000 binary tasks, drawing α from a mixture of high‑skill and low‑skill distributions and β from a range of difficulties. Second, a real‑world dataset collected via Amazon Mechanical Turk on an image classification task (e.g., CIFAR‑10 style) is used to validate performance under realistic worker heterogeneity. All methods are given the same labeling budget B, and three baselines are compared: random selection of worker‑task pairs, a “traversal” baseline that sequentially picks tasks and randomly picks workers, and the proposed active‑learning approach.

Results show that, for an equal number of acquired labels, the proposed method achieves 3–5 % higher overall annotation accuracy than the baselines. Moreover, the correlation between estimated expertise α̂ and ground‑truth α (measured by Spearman’s ρ and Pearson’s r) improves by 0.25–0.35 points, indicating more reliable ranking of workers. Importantly, the active‑learning strategy reaches target accuracies (≈85 %) while using 30 %–50 % fewer labels, demonstrating substantial cost savings. Among worker‑selection policies, always picking the current best worker (Best‑Worker) yields the highest immediate accuracy, while ε‑greedy provides a modest benefit in long‑term expertise estimation by preventing the model from over‑committing to a possibly mis‑estimated worker.

The paper’s contributions are twofold. First, it demonstrates that embedding a probabilistic crowdsourcing model within an active‑learning loop can dramatically reduce labeling costs without sacrificing, and indeed often improving, classification performance. Second, it highlights the importance of jointly considering task uncertainty and worker reliability when deciding which annotation to request, offering a practical framework for real‑world crowdsourcing pipelines where budgets are tight and label noise is inevitable.

Future directions suggested include extending the binary‑label formulation to multi‑class problems, incorporating variable per‑worker labeling costs (e.g., expert vs. novice rates), handling delayed or asynchronous responses, and integrating deep feature representations so that the active‑learning criterion can also exploit feature‑space uncertainty. Overall, the work provides a solid methodological bridge between crowdsourced label aggregation and cost‑effective active learning.