Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability

📝 Abstract

Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams - familiarity with other members - stands in contrast to crowd work’s flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work.

💡 Analysis

Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams - familiarity with other members - stands in contrast to crowd work’s flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work.

📄 Content

Huddler: Convening Stable and Familiar Crowd Teams
Despite Unpredictable Availability Niloufar Salehi, Andrew McCabe, Melissa Valentine, Michael Bernstein Stanford University {niloufar, msb}@cs.stanford.edu, {amccabe, mav}@stanford.edu

ABSTRACT Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams — familiarity with other members — stands in contrast to crowd work’s flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work. Author Keywords Crowdsourcing; crowd work; crowd teams ACM Classification Keywords H.5.3 Group and Organization Interfaces: Collaborative computing, Computer supported cooperative work INTRODUCTION Crowdsourcing achieves impressive goals today by distributing work among independent individuals [6], but
its future success will require collaborative crowd teams. Existing crowdsourcing techniques execute complex work [28] via pre-structured microtask workflows [6, 34]. However, seminal research from the field of organizational design has established that structured workflows are fundamentally incompatible for complex work [3, 50]. This literature suggests that completing complex work under uncertainty requires team-based coordination: teams iteratively establish a course of action, execute it, and then reflect and revise it based on their progress [21, 50]. Recognizing this, complex and creative goals such as design prototyping have prompted systems that assemble ad-hoc teams from the crowd [36, 37, 46]. Unfortunately, the on-demand nature of crowdsourcing would seem to make team-based coordination infeasible. Successful team-based coordination requires that team members build familiarity by working together repeatedly over time [15, 26, 45]. Familiar teams outperform ad-hoc teams by building common ground, learning to coordinate, and utilizing each person’s unique skills [33, 45]. To reap these benefits, teams must keep the same members over time. However, stable team membership is not a core characteristic of crowd work: crowd workers on platforms such as Amazon Mechanical Turk (AMT) are available at unpredictable times [38] and often engage with other tasks when a new opportunity arises. Ad-hoc crowd teams on AMT feature an ever-changing roster of members (e.g., [46]), making it infeasible to build familiarity, and inhibiting the crowd’s ability to achieve complex work. In this paper we present Huddler, a system that enables assembly of familiar crowd teams, even under unpredictable availability and strict time constraints. With Huddler, crowd workers align themselves with any number of teams and request the appropriate team when they accept a task. If a team member is unavailable, Huddler recruits an alternative who maximizes team familiarity, as measured via the number of tasks previously completed with current team members. Huddler’s crowd teams thus maintain a stable core, and bring in familiar faces as peripheral replacements. Huddler must recruit these members under a strict time limit, with knowledge that many workers will not respond or will decline. The system thus modulates its recruitment by measuring how likely a worker is to respond within a given time limit. Planning who to ask, and how long to wait for them before moving on, is a combinatorial problem with an exponential number of possible alternatives. We introduce a dynamic programming algorithm that allows Huddler to compute an optimal recruitment plan under real-time performance constraints.
We first demonstrate that familiarity improves the performance of teams of crowd workers. This effect is known for organizational work teams, but the lack of face- to-face interaction and shared organizational context may make it harder for crowd teams to reap the same benefits of Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the

This content is AI-processed based on ArXiv data.