When Scientists Become Social Scientists: How Citizen Science Projects Learn About Volunteers

Online citizen science projects involve recruitment of volunteers to assist researchers with the creation, curation, and analysis of large datasets. Enhancing the quality of these data products is a fundamental concern for teams running citizen science projects. Decisions about a project’s design and operations have a critical effect both on whether the project recruits and retains enough volunteers, and on the quality of volunteers’ work. The processes by which the team running a project learn about their volunteers play a critical role in these decisions. Improving these processes will enhance decision-making, resulting in better quality datasets, and more successful outcomes for citizen science projects. This paper presents a qualitative case study, involving interviews and long-term observation, of how the team running Galaxy Zoo, a major citizen science project in astronomy, came to know their volunteers and how this knowledge shaped their decision-making processes. This paper presents three instances that played significant roles in shaping Galaxy Zoo team members’ understandings of volunteers. Team members integrated heterogeneous sources of information to derive new insights into the volunteers. Project metrics and formal studies of volunteers combined with tacit understandings gained through on- and offline interactions with volunteers. This paper presents a number of recommendations for practice. These recommendations include strategies for improving how citizen science project team members learn about volunteers, and how teams can more effectively circulate among themselves what they learn.

💡 Research Summary

This paper investigates how the team behind the Galaxy Zoo citizen‑science project learned about its volunteers and how that knowledge shaped project design, operation, and data quality. The authors begin by situating citizen science within contemporary challenges of big‑data management and public engagement, noting that platforms such as Zooniverse promise to alleviate both by recruiting large numbers of volunteers to process, curate, and analyze scientific datasets. A literature review highlights that while technical solutions (e.g., data cleaning, validation algorithms) are essential, effective volunteer management—recruitment, retention, training, and credit mechanisms—is equally critical for producing high‑quality scientific outputs.

The study addresses two research questions: (1) What do project team members learn about volunteers over the life of a citizen‑science project? (2) How do they acquire that knowledge? To answer these, the authors conduct a qualitative case study of Galaxy Zoo’s first four years (July 2007–July 2011). Methods include long‑term ethnographic observation of the website and forums, semi‑structured interviews with 13 team members (astronomers and software engineers) and 14 volunteers, and analysis of project documents, publications, and promotional materials. Interview transcripts are coded using grounded‑theory techniques.

Findings are organized around three pivotal “learning instances.”

1. League‑table experiment – Early in the project a public leaderboard displayed the top 25 classifiers. Within months the team observed that the table encouraged a small subset of highly active volunteers to prioritize speed over accuracy, and demotivated the majority who felt their contributions were insignificant. Evidence came from volunteer emails complaining about “bots,” forum discussions expressing frustration, and interview statements describing three volunteer groups (competitive, demoralized, indifferent). Recognizing the negative impact on data quality and participant diversity, the team removed the leaderboard in October 2007 and shifted toward more inclusive metrics.
2. Serendipitous discoveries – Volunteers identified novel astronomical objects (e.g., “Hanny’s Voorwerp”), demonstrating that participants possess domain‑relevant curiosity and expertise. The team responded by formalizing a discovery‑reporting pipeline, granting credit, and publishing papers that listed volunteers as co‑authors. This reinforced volunteer motivation, legitimized citizen contributions, and contributed to the project’s scientific reputation.
3. Simulated image testing – The team introduced synthetic galaxy images to assess classification bias and to calibrate the interface. By comparing volunteer classifications of real versus simulated data, they quantified systematic errors and iteratively refined training materials and UI cues. This data‑driven feedback loop exemplifies how learning about volunteer performance can directly improve dataset reliability.

Across these instances, the Galaxy Zoo team integrated heterogeneous information sources: quantitative metrics (classification counts, accuracy rates), formal surveys, informal forum posts, direct email feedback, and face‑to‑face/Skype interviews. The authors note that early on, insights were siloed; different sub‑teams (software, science, outreach) often operated with incomplete knowledge of volunteer behavior. To address this, the paper recommends establishing structured knowledge‑sharing mechanisms—regular cross‑disciplinary meetings, internal wikis, and documented case studies—to ensure that learning is rapidly disseminated and incorporated into decision‑making.

The paper concludes with practical recommendations for citizen‑science projects: (a) systematically collect both formal and informal data about volunteers; (b) continuously monitor how design elements (leaderboards, credit systems, UI features) affect motivation and data quality; (c) embed mechanisms for rapid circulation of insights across all team members; and (d) treat volunteer learning as an ongoing, socially‑scientific activity rather than a one‑off survey. By adopting these practices, projects can improve dataset quality, sustain diverse volunteer participation, and enhance the overall impact of citizen science.