Crowdsourced science: sociotechnical epistemology in the e-research paradigm

Synthese DOI 10.1007/s11229-016-1238-2 Cro wdsourced science: sociotechnical epistemology in the e-resear ch paradigm Da vid W atson 1 · Luciano Floridi 2 Receiv ed: 30 April 2016 / Accepted: 29 September 2016 © The Author(s) 2016. This article is published with open access at Springerlink.com Abstract Recent years hav e seen a surge in online collaboration between experts and amateurs on scientific research. In this article, we analyse the epistemologi- cal implications of these cro wdsourced projects, with a focus on Zooniv erse, the world’ s largest citizen science web portal. W e use quantitativ e methods to ev aluate the platform’ s success in producing large volumes of observation statements and high impact scientific discov eries relativ e to more con ventional means of data process- ing. Through empirical evidence, Bayesian reasoning, and conceptual analysis, we sho w how information and communication technologies enhance the r eliability , scal- ability , and connectivity of cro wdsourced e-research, giving online citizen science projects powerful epistemic advantages over more traditional modes of scientific in vestigation. These results highlight the essential role played by technologically mediated social interaction in contemporary kno wledge production. W e conclude by calling for an explicitly sociotechnical turn in the philosophy of science that com- bines insights from statistics and logic to analyse the latest dev elopments in scientific research. K eywords Bayesian confirmation theory · Citizen science · Epistemic logic · Information and communication technology (ICT) · Philosophy of information · Social epistemology · Zooniv erse B David W atson d.watson@qmul.ac.uk 1 Queen Mary Univ ersity of London, London, UK 2 Oxford Internet Institute, Univ ersity of Oxford, Oxford, UK 123 Synthese 1 Introduction Experts and amateurs have been collaborating on so-called ‘citizen science’ projects for more than a century ( Silverto wn 2009 ). Traditionally , such projects relied upon v ol- unteers to participate in data collection . In more recent years, the spread of information and communication technologies (ICTs) has allo wed users to become increasingly in volved in data analysis . Early online citizen science initiati ves made use of par- ticipants’ spare processing power to create distributed computing networks to run simulations or perform other complex functions ( Anderson et a l. 2002 ). The latest wa ve of citizen science projects has replaced this passiv e software approach with inter - activ e web platforms designed to maximise user engagement. Utilising fairly simple tools provided by well-designed websites, amateurs hav e helped model complex pro- tein structures ( Khatib et al. 2011a , b ), map the neural circuitry of the mammalian retina ( Kim et al. 2014 ), and discov er ne w astronomical objects ( Lintott et al. 2009 ; Cardamone et al. 2009 ). As of December 2015, citizen science project aggregator SciStarter links to ov er a thousand activ e projects ( SciStarter 2015 ). What are the philosophical implications of this new brand of crowdsourced e- research? Sociologists hav e studied the demographics and motiv ations of virtual citizen scientists for years (e.g., Nov et al. 2011 ; Rotman et al. 2012 ; Raddick et al. 2013 ), while data scientists hav e extensi vely examined the mechanics of user contri- butions to such sites (e.g., Kawryko w et al. 2012 ; Ponciano et al. 2014 ; Franzoni and Sauermann 2014 ). Philosophers, howe ver , have so far been silent on these methodolog- ical dev elopments. In this article, we argue that a close examination of crowdsourced e-research reveals important lessons for epistemology and philosophy of science. V irtual citizen science labs constitute large sociotechnical systems in which profes- sionals, volunteers, and digital technologies come together to pursue three important epistemic goals: (1) Reliability The designers of citizen science websites employ numerous quality control measures to ensure t hat user contributions are accurate and precise. (2) Scalability Hundreds of thousands of volunteers from around the world regularly participate in citizen science projects, analysing unprecedented volumes of data for a wide v ariety of scientific studies. (3) Connectivity Information and communication networks unlock the distrib uted kno wledge of large epistemic communities by establishing numerous channels that allow users to confer with one another and direct information toward one or se veral central nodes. In what follo ws, we present empirical evidence that crowdsourced e-research is uniquely reliable, scalable, and connectiv e. W e argue that these properties are essential for the promotion of scientific knowledge, and therefore that any system that max- imises all three not only constitutes a major methodological advancement, but merits close philosophical attention. W e conclude that the success of virtual citizen science underscores the irreducibly sociotechnical nature of all scientific inquiry . Follo wing an overvie w of this paper’ s methods in Sect. 2 , we proceed to examine the structural mechanics of contemporary citizen science in Sects. 3 – 5 , with an emphasis on the epistemic advantages afforded by web-enabled mass collaboration. Our findings 123 Synthese indicate that such projects tend to generate more observ ations and higher quality discov eries than similar studies using traditional methods. The significance of these results goes far beyond the limits of citizen science. W e close in Sect. 6 with a revie w of our findings and a proposal for further research in sociotechnical epistemology . 2 Motivation and methods Suppose Albert and Niels are rational agents with opposing views on which of two mutually incompatible scientific hypotheses is correct. Let us assume that fundamental disagreements between the two men are negligible—the y play by roughly the same epistemic rules and are each willing to concede a point in the face of suf ficient evidence or compelling arguments. Y et despite their concordance on basic principles, they just cannot seem to agree on this particular case. What might explain this (common) scenario? Say Niels happens to be right in this instance. Then at least one of three possi- bilities accounts for his success: (a) he got lucky; (b) he had better evidence; or (c) he had a better understanding of the evidence. If our goal is to find the most fruitful strategies for scientific inquiry , then explanation (a) is irrele v ant. Options (b) and (c) are more interesting, howe ver . The first highlights the importance of good evidence, which can be split into data quality and quantity ( Floridi and Illari 2014 ). The second suggests that ev en in the face of identical e vidence, superior results are achiev ed by the agent who does a better job of finding the underlying structure behind a giv en set of observ ations. In one of the seminal works of social epistemology , Goldman ( 2003 ) sets out to e valuate various systems for making and improving judgments through dif ferent forms of testimony . Central to his project is the notion of ‘veritistic v alue’, a measure of one’ s degree of knowledge or truth possession with respect to a proposition. Let T stand for the truth-value function such that T ( p ) = 1i f f p is true and T ( p ) = 0i f f p is false. Let C stand for agent A ’ s credence function such that C A ( p ) = 1i f f A is certain that p and C A ( p ) = 0i f f A is certain that ∼ p . Then the veritistic value of A ’ s judgment with respect to p may be defined as a function V such that V A ( p ) = 1 −| T ( p ) − C A ( p ) | . 1 In our moti vating example abov e, Niels’ judgment was of higher veritistic v alue than Albert’ s. W e submit that for a wide array of projects throughout the natural sciences, cro wd- sourcing offers the best av ailable method for maximising the expected veritistic value of researchers’ hypotheses. Thoughtful web protocols and global Internet access ensure high data quality and quantity , while the sociotechnical network’ s topology pushes anomalous observations to the fore, thereby challenging experts to find the latent structure underlying the natural phenomena they study . This conclusion is derived from a combination of empirical findings and logical reasoning presented below . For the former , we draw primarily on data f rom and about Zooni verse, the world’ s largest citizen science web portal. For the latter , we adopt a Bayesian confirmation theoretic 1 Goldman does not use these precise formulae, although they are implicit in his ‘trichotomous scheme’. See Goldman ( 2003 , Sect. 3.4, pp. 87–94). 123 Synthese frame work that borrows from the social epistemology of Goldman ( 2003 ) and the epistemic logic of Fagin et al. ( 1995 ). Because a priori reflection alone is insufficient to substantiate our argument, we re view Zooniv erse’ s 2014 transaction logs and complete publication history to better understand the platform’ s internal mechanics and scientific output. With dozens of activ e projects and over 1.4 million subscribers worldwide, Zooniverse exemplifies the reliability , scalability , and connectivity of contemporary crowdsourced e-research we intend to analyse. The site began in July 2007 with a single project, Galaxy Zoo, which invited users (aka ‘Zooites’) to help classify the morphological properties of galaxies captured by the Sloan Digital Sky Survey (SDSS). Following the success of this inaugural venture, administrators (aka ‘Zookeepers’) expanded the site into a multi-project platform in December 2009. While the vast majority of Zooniv erse projects are dev oted to topics in the natural sciences, the site has recently branched out to include digital humanities initiativ es as well. Figure 1 provides a breakdown of all 27 projects hosted by the platform in 2014. Unlike the competitiv e games of fered by designers of other popular crowdsourced science sites such as FoldIt ( Cooper et al. 2010 ) and EyeW ire ( Kim et al. 2014 ), Zooniv erse projects are based entirely upon classifications , be they of galaxies, whale calls, or ancient manuscripts. Each project starts with a simple set of instructions on how to classify the relev ant digital artefacts, follo wed by a steady stream of raw data ready for processing ( Simpson et al. 2014 ). As of December 2015, Zooniverse classifications have been the basis for 81 articles published in peer-revie wed journals, in addition to a handful of conference papers and book chapters ( Zooniverse 2015 ). W e examined those publications for content and scientometric performance using Else vier’ s Scopus database and the Thomson Reuters Institute for Scientific Informa- tion Journal Citation Reports. W eb analytic data from Zooni verse’ s 2014 transaction logs were generously provided by the platform’ s administrators. T ogether , these sources provide the empirical basis for this paper’ s epistemological claims. Quan- titativ e analysis was conducted in the R statistical en vironment (version 3.2.2), with significance levels for all tests uniformly fixed at α = 0 . 05. 3 Reliability: the wisdom of the cro wd The success of any scientific study , crowdsourced or otherwise, crucially relies upon the reliability of its observ ations. How can we trust Zooniv erse’ s classification data if they merely represent the uninformed opinion of a large community of amateurs? 3.1 Quality control protocols The insight that groups are often better at producing kno wledge than individuals is an old one. A formal proof of the claim was originally deriv ed by Condorcet ( 1785 ), whose famous jury theorem states that given a defendant of uncertain guilt and a collection of jurors whose judgments are each better than random b ut less than perfect, the majority of jurors is alw ays more likely to be correct than any individual juror . Moreover , the probability of a correct majority judgment approaches 1 as the jury size increases. An 123 Synthese Fig. 1 Dendrogram depicting the typological breakdown of all 27 Zooniverse projects active in 2014 important corollary to Condorcet’ s jury theorem, howe ver , is that opposite results will hold for worse than random jurors. That is, gi ven a jury composed of individuals with a less than 0.5 chance of making accurate judgments, the majority is always more likely to be wrong than any individual juror , and the probability of a correct majority judgment approaches 0 as the jury size increases. The initial sceptical challenge to citizen science is motiv ated by something like the corollary to Condorcet’ s jury theorem ( Collins 2014 ). Highly specialised subjects within the natural sciences are dominated by experts for good reason. Amateur views on particle physics or microbiology are probably wrong, these sceptics allege, and large groups of amateurs pooling their collectiv e ignorance will surely do no better . The issue is perhaps best understood as a special case of the more general problem of testimony , upon which much of social epistemology turns. W ith hundreds of thousands of users participating in any giv en cro wdsourcing project, odds are that at least some will perform worse than random at certain data classification tasks. T o stay on the right side of Condorcet’ s jury theorem, Zooniv erse’ s administrators employ sev eral strategies: • Design simplicity Before a project is launched, Zookeepers ensure that tasks are simple and clearly explained to maximise potential contributors and minimise user error . • Automated filtering Once a project is underway , algorithms filter classifications by user performance and community agreement across observations. 123 Synthese Fig. 2 A screenshot from the original Galaxy Zoo website • Compr ehensive r eview Once a project is completed, classifications are weighted according to each user’ s tendency to be in the majority and full datasets are subject to expert re view . T ogether , these quality control measures have a profound impact on the scientific utility of amateur observations. T o see ho w , consider the case of Galaxy Zoo. Zooniv erse’ s first project was a straightforward classification task, explained to ne w users in a brief tutorial that made no use of technical terminology . V olunteers were presented with paradigmatic examples of standard galaxy types and asked to determine to which type subsequent galaxies properly belonged. A total of six classifications were possible, with small schematic symbols of the av ailable options permanently visible at the right of the screen (see Fig. 2 ). No experience in astrophysics was presumed, and in fact, with just a little practice, even young children could (and did) participate ( Raddick et al. 2013 ). Once users completed the tutorial, they were unknowingly subject to a probationary period during which they were presented with test data that Zookeepers considered unambiguous cases of their particular galactic morphologies. Classifications by those who failed to correctly identify 11 out of their first 15 images were not sav ed in the site’ s database ( Lintott et al. 2008 ). This ensured that erroneous results from volunteers who misunderstood the instructions, experienced technical difficulties, or perhaps e ven maliciously sought to corrupt Galaxy Zoo’ s data, would not confound the project’ s findings. As a further precaution, Zooniverse administrators designed a redundant website architecture in which numerous users revie wed each galaxy before it entered the project’ s catalogue. Objects were processed an average of 38 times each, allowing researchers to estimate the confidence of their conclusions by ev aluating the extent of community consensus around particular classifications ( Lintott et al. 2008 ). Once the entire SDSS surve y had been classified, Zookeepers applied a weighted voting schema in which each user’ s contributions were valued in proportion to the 123 Synthese av erage popularity of her classifications. 2 A comparison of weighted and unweighted results rev ealed that, while there was practically no dif ference between the two scoring methods in terms of ultimate morphological selections, weighting user votes pushed tens of thousands of galaxies past researchers’ 80 and 95 % consensus thresholds for entrance into ‘clean’ and ‘superclean’ morphological samples, respectiv ely ( Lintott et al. 2008 ). A final, crucial step in Zooniv erse’ s quality control protocol is the expert revie w of user observ ations. Examining Galaxy Zoo’ s results, researchers found significant over - classification of anti-clockwise spirality , probably due to the population’ s preference for right handedness ( Land et al. 2008 ). Elliptical galaxies were also over -classified, most likely because spiral galaxies viewed at great distances undergo redshifts that render their arms blurry and hard to detect ( Bamford et al. 2009 ). Land et al. and Bamford et al. were both able to identify these errors and correct for user biases by means of fairly simple algorithms. W ith these measures in place, Galaxy Zoo’ s output exceeded all expectations. Com- paring the project’ s classifications with those from three visual inspection studies conducted by professional astronomers on samples from the same SDSS images, Lintott et al. ( 2008 ) found that Zooites agreed with the experts in more than 90 % of cases—a rate comparable to experts’ mutual agreement with one another . Simi- larly positive results ha ve been reported for numerous crowdsourced projects across the natural sciences, including NASA ’ s Clickwork ers initiativ e ( Kanefsky et al. 2001 ), Stardust@home ( Méndez 2008 ), Foldit ( Khatib et al. 2011b ), and EyeW ire ( Kim et al. 2014 ). It should perhaps come as no surprise to learn that large epistemic communities are capable of generating reliable observations for scientific research. After all, pro- fessors ha ve long relied on untrained undergraduates for basic data collection tasks. The differences between that f amiliar case and this novel one are twofold. In the univ ersity setting, there are academic and social incenti ves to be a proficient data collector . In crowdsourced e-research, the data analysis platform itself ensures user performance. Second, we kno w by Condorcet’ s theorem that a jury’ s verdict asymp- totically approaches truth as the number of better than random jurors increases. The quantity of participants in volv ed in a giv en study therefore has a qualitativ e impact on the judgments they issue. The combination of shre wd web design and sheer user volume can turn the public into a valuable resource for scientific research. 3.2 V eritistic value and Bayesian reasoning Goldman’ s social epistemology relies hea vily on Bayesian inference, a methodology he argues is supported by the veritistic approach. He reports a result he credits to Shaked (see Goldman and Shaked 1991 ), who combines Bayes’ theorem with Jensen’ s 2 The system worked as follo ws. For each galaxy x to which volunteer k assigned morphology F ,t h e partial weight of k ’ s vote was defined as the number of other Zooites who agreed that Fx , di vided by the total number of galaxies classified by k , N x ( k ) . The summation of such ratios for all N x ( k ) represents k ’s total weight w k . T otal weights for all users were then scaled to a mean of 1, and applied to each vote in the database. See Lintott et al. ( 2008 ). 123 Synthese inequality to prove (roughly) that if agent A has an accurate model of hypothesis h , then updating her beliefs with some relev ant e vidence e will tend to bring A closer to h ’ s truth-v alue. Specifically , he shows that, if the following three criteria are met: (1) Relev ance: P ( h )  = P ( h | e ) (2) Bounds: 0 < C A ( h )< 1 and P ( e | h ) P ( e |∼ h )  = 1 (3) Model accuracy: C A ( h ) = P ( h ) and C A ( e | h ) C A ( e |∼ h ) = P ( e | h ) P ( e |∼ h ) then A ’ s expected change in veritistic value after conditionalising upon e is strictly positi ve. 3 That is, E [ V A ( h ) | e − V A ( h ) ] > 0 . Shaked’ s theorem is rather trivial in most applications. Rarely do we ha ve precise v alues of prior probabilities or rele v ant Bayes factors, and if we did, it would hardly be surprising to learn that combining the two would likely produce a net kno wledge increase. Nev ertheless, the result is important in the present context because, as we shall argue, it provides a firm logical foundation for cro wdsourcing in the natural sciences. Say h stands for some particular observational claim, e.g. ‘Galaxy x is elliptical’, and e stands for a set of weighted user votes with respect to galaxy x ’ s morphology . When astrophysicist A examines the data, she is in a good position to e v aluate both the prior probability that x is elliptical, given her background knowledge about the frequency of elliptical galaxies, and the likelihood ratio that x is elliptical, giv en the degree of community consensus e vident in e and/or rele v ant user biases. Even if the quality of user contributions to some particularly confounding project were relativ ely lo w , as long as experts could determine their accuracy , then Shaked’ s theorem prov es that Bayesian reasoning from such data will tend to increase the veritistic value of collectiv e classifications. When amateur testimony is both accurately ev aluated and generally reliable, as the protocols outlined above are designed to ensure, then the resultant data should be of extremely high quality . 3 Our notation differs from that presented by Goldman and Shaked, but the substance of their theorem remains unchanged. See Goldman and Shaked ( 1991 ). Their complete proof only appears in the appendix of a later book, which includes a reprinted edition of Goldman and Shaked’ s original article. See Goldman ( 1992 , chapter 12). 123 Synthese Fig. 3 Log–log scatterplot of Zooniv erse users versus classifications, with an ordinary least squares regres- sion line fit to the data 4 Scalability: the mor e the Merrier High-throughput techniques across the natural sciences hav e gi ven modern researchers more accurate, precise, and numerous measurements than e ver before, yet pattern recognition software for visual, audio, and video data is still fairly crude. How can scientists take advantage of these emerging technologies most efficiently? 4.1 Users and observations in Zooniverse In the months preceding the launch of Galaxy Zoo, Zooniv erse cofounder Ke vin Schawinski spent a week classifying 50,000 galaxies as part of his D.Phil. research in the Astrophysics Department at the Uni versity of Oxford ( Schawinski et al. 2007 ). The task was gruelling. Presuming 12-hour workdays, Schawinski must have aver - aged a classification ev ery six seconds for sev en straight days. By comparison, the day Galaxy Zoo went online, users were av eraging 70,000 classifications per hour ( Nielsen 2011 ). By the time Zooites finished processing the complete SDSS survey of almost 900,000 objects, their work constituted the largest morphological catalogue in the history of astronomy ( Bamford et al. 2009 ). Zooniv erse’ s 2014 transaction logs reveal a strong positive correlation between a project’ s user totals and the number of classifications it generates. Figure 3 is a log–log scatterplot depicting the relationship between these two variables over the 223 complete project-months for which such data were recorded. A simple linear regression model was fit to the log transform data, indicating that user totals account for 123 Synthese approximately 80 % of the variance in a project’ s classificatory output. While variables like user engagement and media coverage would no doubt help to construct a more complete picture of ho w and why some citizen science initiati ves are more fruitful than others ( Cox et al. 2015 ), this plot clearly shows that the number of volunteers who contribute to a project is a strong predictor of how many observations it will produce. The success of any gi ven citizen science project has always been dependent on its ability to attract sufficient volunteers. Ho wev er , only in the era of global ICT networks can these initiatives reach the critical mass at which they begin to match or ev en surpass the efforts of professionals relying on more traditional modes of data processing. Consider , for example, the case of astronomical catalogues. An astronomical catalogue is a complete list of objects of some common type (e.g., galaxies) detected by one or several instruments working in concert, usually as part of an astronomical survey (e.g., the SDSS). While scientific articles often draw on select or simulated data to explore some particular phenomenon, astronomical catalogues represent researchers’ total observational output of a particular kind. Comparing the number of observations in traditional and cro wdsourced editions of such works therefore offers the best means of testing the relativ e fruitfulness of the two methodologies. In the four years since the aforementioned Galaxy Zoo catalogue was published, Zooniv erse has gathered user classifications into sev en more astronomical catalogues, two of which were the first of their kind. 4 The other five are the largest of their sort e ver compiled, exceeding pre vious record holders by more than order of magnitude on average. By comparison, traditional catalogues tend to build on previous work in increments of about 80 %. T able 1 includes observation totals for each of these five Zooniv erse catalogues 5 and the traditional catalogues the y superseded, 6 along with the percent increases in observation counts represented by each. Where possible, statistics on three previous catalogues are included for comparison. A Kolmogoro v–Smirnov (K–S) test found significant difference between the per- cent increases in observation totals represented by Zooni verse projects and those of traditional catalogues relativ e to previous collections, D = 0 . 86 , p = 0 . 02. While the 4 Follo wing the discovery of a rare object in the initial Galaxy Zoo project (about which more below), Zooniv erse launched an intergalactic search for similar anomalies, ultimately resulting in the identification of 19 candidate ‘voorwerps’ ( Keel et al. 2012 ). Though there are sev eral other coronal mass ejection (CME) catalogues, Zooniverse’ s is unique in that it deliberately prioritises quality over quantity , ignoring minor CMEs while gathering the most extensiv e time series data ever recorded on a relatively small number of notable solar e vents ( Barnard et al. 2014 ). Note that, because neither Zooniverse project bears quantitative comparison with any traditional catalogue, both are excluded from the following analysis. 5 Galaxy Zoo 1 gathered basic galactic morphologies ( Lintott et al. 2011 ); Galaxy Zoo 2 was devoted to detailed galactic morphologies ( Willett et al. 2013 ); results from both projects were used to create a catalogue of overlapping galaxies ( Keel et al. 2013 ); the Milky W ay Project found infrared bubbles in our own galaxy ( Simpson et al. 2012a ); and the Andromeda Project sought stellar clusters in our neighbouring Andromeda galaxy ( Johnson et al. 2015 ). 6 All previous observations of overlapping galaxies are catalogued in Appendix A of ( Keel et al. 2013 ); traditional catalogues of infrared bubbles were compiled by Churchwell et al. ( 2006 , 2007 ); the three largest collections of basic galactic morphologies gathered by traditional means are all due to Schawinski et al. ( 2007 ); Fukugita et al. ( 2007 ), Baillard et al. ( 2011 ), and Nair and Abraham ( 2010 ) used visual inspection to catalogue detailed galactic morphologies of increasing size; and the three largest stellar cluster catalogues compiled before Zooniverse were published by Bastian et al. ( 2012 ), San Roman et al. ( 2010 ), and Popescu et al. ( 2012 ), respectiv ely . 123 Synthese Ta b l e 1 Observation totals and percent increases across five different types of astronomical catalogues Catalogue Method Observations % Increase Overlapping Galaxies T raditional 25 Crowdsourcing 1990 7860 Infrared Bubbles T raditional 322 T raditional 591 83.54 Crowdsourcing 5106 763.96 Basic Galactic T raditional 15 , 729 Morphologies T raditional 19 , 649 24.92 T raditional 48 , 023 144.40 Crowdsourcing 738 , 175 1437.13 Detailed Galactic Traditional 2253 Morphologies T raditional 4458 97.87 T raditional 14 , 034 214.80 Crowdsourcing 304 , 122 2067.04 Stellar Clusters T raditional 751 T raditional 803 6.92 T raditional 920 14.57 Crowdsourcing 2753 199.24 sample size in this analysis is admittedly small, the ef fect size detected is very large, Cohen’ s d = 1 . 22, demonstrating a difference of more than a full standard deviation between the two groups’ means. Giv en the strength and uniformity of these results, we may confidently conclude that crowdsourcing is categorically superior to tradi- tional visual inspection methods at gathering large quantities of empirical evidence for astronomical studies. Similar results have been reported for large-scale ecology projects ( Swanson et al. 2015 ). 4.2 Epistemic communities and the principle of total evidence The plot in Fig. 3 suggests that observations are a monotonically increasing function of users in Zooniv erse. Note that the deliberate redundancy mentioned in Sect. 3.1 , whereby each datum is classified numerous times by various users, has no bearing on the regression line’ s slope or residual error . The only parameter subject to change, should all values of the dependent variable be divided by some constant (say , 38), would be the line’ s intercept, as the data points would all shift downward with no impact on the model’ s goodness of fit. This direct relationship between a project’ s contributors and its data processing power is strong evidence in favour of crowdsourcing’ s scientific utility . As we have seen, the largest astronomical catalogues ev er collected were made with the assistance of hundreds of thousands of volunteers. The value in maximising relev ant data for empirical analyses is widely recognised, though rarely does the practice receive explicit justification. Bernoulli ( 1713 )w a s 123 Synthese perhaps the first to write that probability calculations require the use of all a vailable e vidence. K eynes built upon this view , arguing that, while new observations may raise or lo wer the likelihood of a giv en hypothesis, the y in variably increase what he called ‘the weight of e vidence’, leading to ‘more substantial’ conclusions ( 1921 , p. 77). 7 Carnap upgrades this proposal to a full blown principle, claiming that ‘In the application of inductiv e logic to a giv en knowledge situation, the total evidence av ailable must be taken as basis for determining the degree of confirmation’ ( 1950 , p. 221). Though some have challenged Carnap on this point ( A yer 1957 ; McLaughlin 1970 ), the vast majority of philosophers, statisticians, and laypeople alike tend to vie w the principle of total e vidence (TE) as little more than common sense ( Hempel 1960 ; Efron 2010 ). There are se veral compelling reasons to accept TE. Increased sample sizes improv e the accuracy and precision of statistical estimates and inferences, narrowing the con- fidence intervals around predictions and parameters, thereby limiting the likelihood of T ype I and T ype II errors. The epistemological merits of TE can be formalised in a Bayesian framework using Shaked’ s theorem. Let e stand for some collection of observ ations, say of galactic morphologies. Let e * stand for some larger body of similar observ ations, say twice as many galactic morphologies. Let h stand for some rele vant hypothesis, perhaps pertaining to the distribution of galactic morphologies. Then while e *’ s superior weight alone does not entail any conclusions regarding the relativ e values of the conditional probabilities P ( h | e ) and P ( h | e ∗ ) , we can be more confident in the latter ev aluation than in the former . It follows from Shaked’ s theorem that heavier bodies of e vidence will tend to increase the veritistic value of our judgment in h . Provided the following modified conditions are met: (1) Relev ance: P ( h )  = P ( h | e ) and P ( h )  = P ( h | e ∗ ) (2) Bounds: 0 < C A ( h )< 1 , P ( e | h ) P ( e |∼ h )  = 1 , and P ( e ∗| h ) P ( e ∗| ∼ h )  = 1 (3) Model accuracy: C A ( h ) = P ( h ), C A ( e | h ) C A ( e |∼ h ) = P ( e | h ) P ( e |∼ h ) , and C A ( e ∗| h ) C A ( e ∗| ∼ h ) = P ( e ∗| h ) P ( e ∗| ∼ h ) then what holds for prior and posterior probabilities in Shaked’ s theorem will hold for beliefs updated with e and e *, respectiv ely . That is, we may derive the following inequality: 7 The term ‘weight of evidence’ is employed in a very different sense by Good ( 1983 ), and still another by Joyce ( 2005 ). In what follows, we adopt the Keynesian terminology . See Joyce ( 2005 ) for an insightful breakdown of the subtle distinctions between various interpretations of evidentiary weight, balance, and specificity in Bayesian contexts. 123 Synthese E [ V A ( h ) | e ∗ − V A ( h ) | e ] > 0 . The intuitiv e appeal of TE now becomes clear . An epistemic agent conditionalising upon a relativ ely large collection of observations is more likely to be right about a rele vant hypothesis than she would be giv en a smaller body of similar evidence. This result goes hand in hand with Good’ s theorem ( 1967 ), which purports to prove that rational agents must maximise free evidence, although his argument relies upon extra premises that we do not consider here. Gathering as many observations as possible for scientific inv estigation is not just a matter of fine-tuning particular models. Large samples are more likely t o contain anomalous data, which numerous historians and philosophers of science point out are crucial for theoretical progress. Such unexpected discoveries may falsify prev ailing hypotheses ( Popper 1959 ) or perhaps even help inaugurate a ne w research paradigm ( Kuhn 1962 ). Since anomalous observations are, by definition, low probability ev ents, we should only expect to find them in large datasets. While one or two anomalies could plausibly be dismissed as mere outliers, the accumulation of rare data in large sample sizes makes their presence more salient and their need for explanation more pressing. Giv en the results of the regression in Sect. 4.1 and the preceding defence of TE, it is tempting to conclude that veritistic value is a monotonically increasing function of epistemic community size. Y e t the generality of this claim is constrained by two factors: the nature of a particular scientific in vestigation, and the technology av ailable to those who undertake it. The Zooni verse model is only applicable to projects with intractable amounts of data that require little or no expertise to process. This describes a large and diverse but by no means exhausti ve set of scientific studies. V irtual citizen science also presumes a technological context in which computational resources are sufficiently advanced to establish a global cro wdsourcing platform, but cannot (yet) be used to reliably automate the tasks put forward to volunteers. Numerous groups, including members of the Zooni verse team, are hard at work to create software that will render the user classification system obsolete ( Banerji et al. 2010 ; Simpson et al. 2012b ; Shamir et al. 2014 ). Zookeepers predict that, ev en once such programs are employed, volunteers will remain a v aluable part of e-research, helping to refine algorithms through anomaly detection and re view ( Clery 2011 ; Fortson et al. 2012 ). When it comes to participation in citizen science, the more the merrier . Only online platforms offer the kind of scalability required to host hundreds of thousands of vol- unteers for any given project, and only at these volumes does the data processing po wer of untrained amateurs begin to compete with (or exceed) that of experts using traditional observation methods. The combination of high quality and high quantity data is essential for scientific confirmation and discov ery . 5 Connectivity: E Pluribus Unum The reliability and scalability of crowdsourced e-research has helped amass enormous volumes of reliable observations across the natural sciences. But are the methodol- ogy’ s contributions limited to clev er web design and evidentiary archiving, or does 123 Synthese Other Zooniverse Other Zooniverse Mean Citations Per Article 2008 2009 2010 2011 2012 2013 2014 Median Citations Per Article Year 2008 2009 2010 2011 2012 2013 2014 Year 0 50 100 150 200 0 50 100 150 200 Fig. 4 Bar plots comparing mean and median citations per article for Zooniverse and other sources using the same raw data. Since academic citations are usually po wer-law distributed ( Barabási 2002 ), the median is probably a more reliable measure of central tendency than the mean for these distributions cro wdsourcing hold promise for more substantial forms of scientific knowledge as well? 5.1 Scientometric performance The quality of a scientific discov ery is notoriously difficult to quantify . Ho wev er , the analytic tools of scientometrics provide several methods for attempting to do so ( Price 1963 ; Leydesdorf f 2001 ). Because the majority of Zooniv erse projects draw their raw data from public access archiv es, such as the SDSS and the Hubble Space T elescope, other papers by scientists using the same source materials constitute the most natural control group for scientometric analysis. Of t he 68 Zooniv erse articles published before 2015, 62 were the result of projects that relied exclusi vely on publicly av ailable data. In the same timeframe, other scientists published 5522 articles using the same sources. Comparing t he citation and journal data of these two groups provides some insight into the relative influence of Zooniv erse’ s scientific output. 8 A simple technique of weighing the two samples against each other is through the common scientometric indi- cator of citations per article. This statistic is biased tow ards older articles f or obvious reasons, which accounts for the steep drop off ov er time e vident in Fig. 4 . H ow eve r, both charts reveal another clear trend. W ithout exception, Zooniv erse’ s papers are consistently more cited on ave rage than those by scientists using traditional research methodologies to in vestigate the same material. While the large discrepancy in 2008 8 Because Zooniv erse has been widely studied by sociologists, only citations from natural science journals were counted for this comparison. The true influence of Zooniverse publications in fact extends beyond this narrowly circumscribed academic domain. 123 Synthese Mean: 69.51 Median: 77.66 Mode: 44.83 SD: 25.74 N = 62 Zooniverse Articles Citation Percentile Density 02 0 4 0 6 0 8 0 1 0 0 0.000 0.005 0.010 0.015 0.020 0.025 0.030 Fig. 5 Histogram depicting the distribution of citation percentiles across all Zooni verse articles published from 2008 to 2014. A normal curve N ( 50 , 16 . 67 2 ) is overlaid for comparison, with parameters chosen so as to centre the distribution at the middle of the citation percentile range and let all points under the curve on [0, 100) fall within three standard deviations of the mean is likely due to the substantial buzz around the first Galaxy Zoo article, these bar plots demonstrate that the trend has remained remarkably persistent over time. W e might expect that the citation percentiles by year and data source for a theoretical ‘av erage’ lab would tend to follow an approximately normal distribution, with a small but roughly equal number of articles performing very well and very poorly , and the v ast majority falling somewhere in between. If so, then we can confidently assert that Zooniv erse is not an av erage lab. Fig. 5 is a histogram of Zooniv erse’ s citation percentiles, with a normal curve ov erlaid for comparison. W e find here that nearly half of all Zooni verse papers are in the top quintile of most cited articles for their year and data source, with more than a quarter in the top 10 %. A K–S test found significant de viation between these observed results and those expected of a normal distribution, D = 0 . 48 , p < 0 . 001. The distribution of Zooniv erse’ s citation percentiles has a skewness of γ 1 =− 1 . 04, reflecting a high incidence of papers in the upper ranges of most cited articles for their year and data source. By contrast, the distribution of citation percentiles for the 5522 articles in the control group is nearly uniform. The dissimilar shapes of the two distributions are clearly visualised in Fig. 6 , where density plots for both are ov erlaid f or comparison. W e find here that articles by researchers using traditional methodologies are more concentrated below approximately the 50th percentile, while Zooniv erse papers are more likely t o be found in the upper half of the data range. A K–S test found significant dif ference between the two groups, D = 0 . 35 , p < 0 . 001. Zooniv erse’ s influence is riv alled only by that of the most prestigious labs in the field. Of the 5522 articles in the control group, 136 were published by researchers af fil- 123 Synthese 0 2 04 06 08 0 1 0 0 0.000 0.005 0.010 0.015 Influence of Articles Citation Percentile Density Other Zooniverse 0 2 04 06 08 0 1 0 0 0.000 0.005 0.010 0.015 Influence of Articles Citation Percentile Density Cambridge Zooniverse Fig. 6 Density plots representing the distribution of citation percentiles for Zooniverse articles versus those by all others using the same raw data, and Cambridge researchers using the same raw data, respectively iated with the University of Cambridge, home to one of the most esteemed astronomy institutes in the world. The distribution of citation percentiles for these papers is neg- ativ ely ske wed, γ 1 =− 0 . 43, as one might expect—b ut less so than that of Zooniv erse articles, indicating that the latter are more likely to hav e higher citation percentiles than the former . A K–S test on the two distributions found no statistically significant difference between them, D = 0 . 18, p = 0 . 12, suggesting that Zooniv erse’ s citation percentiles could plausibly represent a random sampling of Cambridge’ s. The disparity in article influence between Zooniverse’ s publications and those from the general population cannot be accounted for by journal data alone. System- atic comparison of the average impact factor 9 and h-index 10 of both groups’ top ten most frequent publishers of articles weighted by output for each year between 2008 and 2014—journals that cumulativ ely account for over 90 % of all such material— demonstrates that Zooniv erse had no systematic advantage in academic visibility to bolster its citation numbers. While the two statistics visualised in Fig. 7 do not perfectly coincide, they both reflect a broadly similar state of aff airs. By either measure, Zooniv erse’ s publishers are roughly as influential as those of other researchers using the same data sources over time. K–S tests on the two pairs of weighted av erages found insignificant dif ferences between the distributions, with Zooniverse’ s journals tending to hav e marginally lower impact factors, D = 0 . 57 , p = 0 . 21, and h-index es, D = 0 . 43 , p = 0 . 54, on average. Of course, the true value of a scientific disco very is impossible to measure. It corresponds to an abstract and subjective concept that ev olves over time and has no clear operationalisation. Howe ver , it is hard to imagine ho w Zooniv erse publications 9 A journal’s impact factor refers to the ratio of its total number of articles cited by other indexed publications within the past two years, and the total number of articles published by that journal in the past two years ( Garfield 1972 ). Impact factor data for 2008–2014 was gathered from the ISI Journal Citation Reports. 10 A journal’ s h-index is defined as its number of articles h that have each been cited in other journals at least h times ( Hirsch 2005 ). H-index data for 2008–2014 was compiled from Else vier’ s Scopus database. 123 Synthese Other Zooniverse Other Zooniverse Average Journal Impact Factor Year 01234567 2008 2009 2010 2011 2012 2013 2014 2008 2009 201 0 2011 2012 2013 2014 Average Journal H-Index Year 0 50 100 150 200 250 Fig. 7 Bar plots comparing the mean impact factor and h-index values for publishers of Zooniverse papers with those by others using the same raw data could so consistently outperform those by other labs in the same field using the same data if they did not at least sometimes contain substanti ve contributions to scientific discourse. This minimal claim is all that is required to answer our question at the top of Sect. 5 in the affirmati ve. Crowdsourcing can and does produce high quality science beyond mere data aggregation. 5.2 Network architecture and distributed knowledge The quality and quantity of observations gathered by Zooniv erse no doubt factors into the strong scientometric performance of their publications over time. Novelty and good publicity may also play a role ( Cox et al. 2015 ). But it is the structure of the site’ s sociotechnical network that truly enables principal in vestigators to harness the community’ s resources for maximal discovery value. Some of Zooni verse’ s most important contributions hav e been the result of confused users taking to the site’ s talk forums to discuss strange objects that did not seem to fit into any of the av ailable categories for classification. That was the case with ‘Hanny’ s voorwerp’, a large cloud of bright green gas in the constellation Leo Minor, which researchers believ e may be the first quasar light echo ever observed ( Lintott et al. 2009 ). User comments also led to the discovery of so-called ‘green pea galaxies’ ( Cardamone et al. 2009 ), triple mergers ( Darg et al. 2011 ), supernov as ( Smith et al. 2011 ), and overlapping galaxies ( Keel et al. 2013 ) in SDSS data. Revie wing the results of their inaugural project, the site’ s founders concluded that ‘The Galaxy Zoo forum has been a scientific gold mine’ ( Fortson et al. 2012 , p. 226). Zooites not only classify the objects provided by Zookeepers, but flag anomalies for further discussion. The intermingling of diverse views and le vels of expertise in the Zooni verse talk forums naturally drives expert attention toward the most deserving data ( Page 2007 ). In sev eral cases, researchers have used those findings to launch new 123 Synthese Fig. 8 Diagram of sociotechnical knowledge production in Zooniv erse. The first four nodes of the network (i.e., every step prior to sending discoveries to a journal) form a recursive loop that results in increasingly refined observational results projects that branch of f from earlier ones in pursuit of similar rare objects. This process demonstrates the new and remarkable ways in which amateurs, experts, and digital technologies come together to form a cohesi ve sociotechnical system in crowdsourced science projects. Figure 8 depicts the knowledge production network in Zooniverse. Note how computer-mediated human cognition at the nodes is transferred by ICTs at the edges, creating a complex epistemic system that refines and curates observations until ready for publication. Such recursive patterns of discovery are i ndicative of a mature and fruitful scientific methodology . The sociotechnical network depicted above is designed to unlock the distributed kno wledge of Zooites and Zookeepers. The formal definition of distributed knowledge was originally proposed by Halpern and Moses ( 1990 ) and later refined by Fagin et al. ( 1995 ). A complete explication of the semantics for their model of epistemic logic is beyond the scope of this paper , but the basic idea is fairly intuitive. Their distributed kno wledge operator D is defined in such a way that, f or some group of agents G , D G represents not only the sum of all things known by G ’ s members, but also all valid entailments of their pooled kno wledge. 11 A version of Fagin et al.’ s logic is widely implemented in multiagent computing ( W ooldridge 2002 ), and has clear applications to any form of collaborativ e research. For instance, it helps defuse the philosophical puzzles that arise when large teams of experts produce results that no single one of them fully understands. Hardwig ( 1985 ) calls this the pr oblem of epistemic dependence , and proposes an elaborate theory of justification in his effort to salvage the primacy of individual knowledge. Longino 11 Say Alice knows that either 3 or 4 is prime. Bob is unsure about 3, but he is certain that 4 is not prime. Then ev en though neither Alice nor Bob alone kno ws that 3 is prime, together they could deduce this fact. The knowledge that 3 is prime is distributed between Alice and Bob, whether they realise it or not. 123 Synthese ( 1990 , 2001 ) challenges Hardwig’ s epistemic individualism, arguing that cognitiv e processes are essentially social, and therefore that individual knowledge itself is either emergent or misconstrued. Neither alternativ e is particularly compelling. Longino’ s account has counterin- tuitiv e consequences for philosophy of mind, while Hardwig’ s appears to be based on a metaphysical misunderstanding. His reticence to grant epistemic agency to an entire research group is probably rooted in the metatheoretical desire for ontological parsimony . If we have already acknowledged the existence of agents A and B , then we would rather avoid countenancing the existence of some third agent C such that C = A ∪ B . It is not entirely clear , howe ver , what metaphysical commitments accom- pany propositions like ‘The jury finds the defendant guilty’, ‘The army won the battle’, ‘The class is on a field trip’, etc. Multiagent systems are regularly treated as perfectly ordinary epistemic ( Goldman 2003 ) and indeed moral subjects ( Floridi 2013 ). Mereological subtleties and confusions abound in the natural sciences, not least because it is often difficult or impossible to establish the ideal unit of analysis ( W inther 2011 ). The question of when to assign collective agency t o a group of individuals raises particularly vexing issues in biology ( Jones 2017 ), not to mention moral phi- losophy ( Searle 1990 ). Some notable philosophers argue that all talk of aggregation is essentially pragmatic, with little or no ontological implications. For example, Hume ( 1748/2008 ) writes that ‘the uniting of…parts into a whole, like the uniting of sev- eral distinct countries into one kingdom, or sev eral distinct members into one body , is performed merely by an arbitrary act of the mind, and has no influence on the nature of things’ (9.11/65). Wittgenstein echoes this sentiment, rejecting the notion that there are any objective distinctions to be drawn between parts and wholes. ‘T o the philosophical question: “Is the visual image of this tree composite, and what are its component parts?” the correct answer is: “That depends on what you understand by ‘composite’.” (And that is of course not an answer but a rejection of the question)’ ( 1953 , § 47). The sociotechnical network may not be a metaphysical entity per se, but its epis- temic agency is explanatorily essential to the knowledge it generates at the system le vel of abstraction ( Floridi 2011 ; W inther 2011 ). The mere aggregation of Zooni- verse’ s units—some users here, a mainframe there—does not begin to account for the site’ s consistent output of high impact scientific publications. It is the complete sociotechnical process, not a summation of localised kno wers, that leads to new and influential discov eries in crowdsourced e-research. Proper coordination is essential ( Floridi 2004 ). Cautious philosophers who accept the notion of distrib uted cognition but balk at the idea of extended or collective agency ( e.g., Giere 2007 ) are insisting on a distinction without a difference. Drawing circles around ev ery indi vidual in volv ed in these projects and declaring that agency can only exist within those borders is as arbitrary as it is unnecessary ( Longino 2013 ). Epistemic agency supervenes upon the people and technology of which the sociotechnical system is comprised, lev eraging both human intelligence and computational resources. Crowdsourcing is hardly the only activity in which this kind of heterogeneous connectivity is evident ( Hutchins 1995 ; Cetina 1999 ; Latour 2005 ), but it does pose a vivid example of how large groups come together to forge scientific kno wledge. 123 Synthese Cro wdsourced science may constitute a radical departure from traditional research methodologies, but its most interesting features lie not in what it adds to scientific inquiry so much as what it r eveals about it. Note how technology permeates ev ery step in the knowledge production chain diagrammed in Fig. 8 . Not only do the arrows depict the flow of information through ICT networks, but at every node people use com- puters to generate, analyse, simulate, and/or disseminate information to other nodes. While epistemologists ov er the last few decades have begun to focus on the social aspects of science, comparativ ely little attention has been paid to its technological underpinnings. The very act of measurement itself, perhaps the most fundamental of all scientific activ ities, requires at least some minimal tools. Especially in the natural sciences, where sophisticated instruments are increasingly operated by computers, simulation has become an essential research methodology , and large groups of collaborators frequently share data via online networks, there can be no denying that technology functions as a mediating, e ven constitutiv e component of epistemic systems. 6 Conclusion Statistical analysis of Zooniv erse’ s publications and user activity indicates that cro wd- sourcing is a uniquely reliable, scalable, and connectiv e method of generating scientific kno wledge. This empirical evidence is supported by Bayesian reasoning within an epistemological framework that seeks to maximise the expected veritistic value of scientific hypotheses. Our work clarifies the philosophical foundations of virtual cit- izen science and highlights the irreducibly sociotechnical component of scientific research. Collaboration and computation are ubiquitous across the natural sciences, and hav e been for decades. The recent popularity of websites like Zooniverse is a salient reminder of how potent the combination of large epistemic communities and well- designed technologies can be. The philosophical implications of this union have not gone completely unremarked (see Cetina 1999 ; Clark 2008 ; Floridi 2011 ), and some recent unpublished doctoral dissertations (e.g., Zollman 2007 ; Simon 2010 ) suggest that it may be a growing area of research. Further in vestigation of science’ s sociotech- nical nature will prov e fruitful for theorists and practitioners alike. W e cannot be certain just what scientific developments the future holds in store, but we can be confident that many of our next great discoveries will be made thanks to some complex partnership of minds and machines. Whether or not such results are the product of crowdsourcing, thorough inv estigation of this strange and remarkable methodology sheds new light on the v aried modes of human knowledge. Clearly the time has come to endorse a sociotechnical turn in the philosophy of science that com- bines insights from statistics and logic to analyse the latest dev elopments in scientific research. Acknowledgments The authors would like to thank David Kinney for his insightful comments on earlier drafts of this article. W e also thank our anonymous referees for their numerous helpful recommendations. 123 Synthese Open Access This article is distributed under the terms of the Creative Commons Attrib ution 4.0 Interna- tional License ( http:// creativ ecommons.org/licenses/ by/ 4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you giv e appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. References Anderson, D. P ., Cobb, J., K orpela, E., Lebofsky , M., & W erthimer, D. (2002). SETI@home: An experiment in public-resource computing. Communications of the A CM , 45 (11), 56–61. A yer , A. J. (1957). The conception of probability as a logical relation. In S. Körner (Ed.), Observation and Interpr etation (pp. 12–30). London: Butterworths. Baillard, A., Bertin, E., de Lapparent, V ., Fouque, P ., Arnouts, S., Mellier, Y ., et al. (2011). The EFIGI catalogue of 4,458 nearby galaxies with detailed morphology . Astr onomy & Astr ophysics , 532 , A74. Bamford, S., Nichol, R. C., Baldry , I. K., Land, K., Lintott, C. J., Schawinski, K., et al. (2009). Galaxy Zoo: the dependence of morphology and colour on environment. Monthly Notices of the Royal Astronomical Society , 393 (4), 1324–1352. Banerji, M., Lahav , O., Lintott, C. J., Abdalla, F . B., Schawinski, K., Bamford, S., et al. (2010). Galaxy Zoo: Reproducing galaxy morphologies via machine learning. Monthly Notices of the Royal Astronomical Society , 406 (1), 342–353. Barabási, A. (2002). Linked: How everything is connected to everything else and what it means for business, science, and everyday life . New Y ork: Penguin. Barnard, L., Scott, C., Owens, M., Lockwood, M., Tuck er-Hood, K., Thomas, S., et al. (2014). The solar stormwatch CME catalogue: Results from the frist space weather citizen science project. Space W eather , 12 (12), 657–674. Bastian, N., Adamo, A., Gieles, M., Silva-V illa, E., Lamers, H., Larsen, S. S., et al. (2012). Stellar clusters in M83: Formation, ev olutions, disruption and the influence of the environment. Monthly Notices of the Royal Astr onomical Society , 419 (3), 2606–2622. Bernoulli, J. (1713). Ars Conjectandi . Basel: Impenfis Thurnisiorum. Cardamone, C., Schawinski, K., Sarzi, M., Bamford, S., Bennert, N., Urry , C. M., et al. (2009). Galaxy Zoo green peas: Discovery of a class of compact extremely star-forming galaxies. Monthly Notices of the Royal Astr onomical Society , 399 (3), 1191–1205. Carnap, R. (1950). Logical foundations of probability . Chicago: University of Chicago Press. Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge . Cambridge, MA: Harvard Uni- versity Press. Churchwell, E., Povich, M. S., Allen, D., T aylor, M. G., Meade, M. R., Babler , B. L., et al. (2006). The bubbling galactic disk. The Astrophysical Journal , 649 (2), 759–778. Churchwell, E., W atson, D. F ., Povich, M. S., T aylor, M. G., Babler , B. L., Meade, M. R., et al. (2007). The bubbling galactic disk. II. The inner 20 ◦ . The Astr ophysical Journal , 670 (1), 428–441. Clark, A. (2008). Supersizing the mind: Embodiment, action, and cognitive extension . New Y ork: Oxford Univ ersity Press. Clery , D. (2011). Galaxy Zoo volunteers share pain and glory of research. Science , 333 (6039), 173–175. Collins, H. (2014). Ar e we all experts now? . Cambridge: Polity . Condorcet, N. (1785). Essai sur l’application de l’analyse à la pr obabilité des décisions rendues à la pluralité des voix . Paris: Imprimerie Royale. Cooper , S., Khatib, F ., T reuille, A., Barbero, J., Lee, J., Beenen, M., et al. (2010). Predicting protein structures with a multiplayer online game. Natur e , 446 (7307), 756–760. Cox, J., Oh, E. Y ., Simmons, B., Lintott, C. J., Masters, K., Greenhill, A., et al. (2015). Defining and measuring success in online citizen science: A case study of zooniverse projects. Computing in Science & Engineering , 17 (4), 28–41. Darg, D. W ., Kaviraj, S., Lintott, C. J., Schawinski, K., Silk, J., L ynn, S., et al. (2011). Galaxy Zoo: Multimergers and the millennium simulation. Monthly Notices of the Royal Astr onomical Society , 416 (3), 1745–1755. Efron, B. (2010). Lar ge scale infer ence . New Y ork: Cambridge Uni versity Press. Fagin, R., Halpern, J. Y ., Moses, Y ., & V ardi, M. Y . (1995). Reasoning about knowledge . Cambridge, MA: MIT Press. Floridi, L. (2004). On the logical unsolvability of the Gettier problem. Synthese , 142 (1), 61–79. 123 Synthese Floridi, L. (2011). The philosophy of information . Oxford: Oxford Uni versity Press. Floridi, L. (2013). Distributed morality in an information society . Science and Engineer Ethics , 19 (3), 727–743. Floridi, L., & Illari, P . (Eds.). (2014). The philosophy of information quality . New Y ork: Springer . Fortson, L., Masters, K., Robert, N., Borne, K. D., Edmonsdon, E. M., Lintott, C. J., et al. (2012). Galaxy Zoo: Morphological classification and citizen science. In M. J. W ay , J. D. Scargle, K. M. Ali, & A. N. Sriv astav a (Eds.), advances in machine learning and data mining for astronomy (pp. 213–236). Boca Raton, FL: T aylor & Francis Group. Franzoni, C., & Sauermann, H. (2014). Crowd science: The organization of scientific research in open collaborativ e projects. Resear ch P olicy , 43 , 1–20. Fukugita, M., Nakamura, O., Okamura, S., Y asuda, N., Barentine, J. C., Brinkmann, J., et al. (2007). A catalog of morphologically classified galaxies from the Sloan Digital Sky Survey: North equatorial region. Astronomical Journal , 134 (2), 579–593. Garfield, E. (1972). Citation analysis as a tool in journal ev aluation. Science , 178 (4060), 471–479. Giere, R. N. (2007). Distributed cognition without distributed knowing. Social Epistemology , 21 (3), 313– 320. Goldman, A. (1992). Liaisons: Philosophy meets the cognitive and social sciences . Cambridge, MA: MIT Press. Goldman, A. (2003). Knowledge in a social world . New Y ork: Oxford University Press. Goldman, A., & Shaked, M. (1991). An economic model of scientific acti vity and truth acquisition. Philo- sophical Studies , 63 (1), 31–55. Good, I. J. (1967). On the principle of total e vidence. The British Journal for the Philosophy of Science , 17 (4), 319–321. Good, I. J. (1983). W eight of evidence: A brief survey . In J. M. Bernardo, M. H. DeGroot, D. V . Lindley , &A .F .M .S m i t h( E d s . ) , Bayesian statistics 2 (pp. 249–270). Oxford: Oxford University Press. Halpern, J. Y ., & Moses, Y . (1990). Knowledge and common knowledge in a distributed environment. Journal of the ACM , 37 (3), 549–587. Hardwig, J. (1985). Epistemic dependence. Journal of Philosophy , 82 (7), 335–349. Hempel, C. (1960). Inductiv e inconsistencies. Synthese , 12 (4), 439–469. Hirsch, J. E. (2005). An index to quantify an individual’ s scientific research output. Pr oceedings of the National Academy of Sciences of the United States of America , 102 (46), 16569–16572. Hume, D. (1748/2008). An enquiry concerning human understanding . Oxford: Oxford University Press. Hutchins, E. (1995). Cognition in the wild . Cambridge, MA: MIT Press. Johnson, L. C., Dalcanton, J. J., Fouesneau, M., W eisz, D. R., W illiams, B. F ., Beerman, L. C., et al. (2015). PHA T stellar cluster survey . II. Andromeda project cluster catalog. Astr ophysical Journal , 802 (2), 127–148. Jones, D. (2017). The biological foundations of action . New Y ork: Routledge. Joyce, J. (2005). How pr obabilities reflect evidence . Philosophical P erspectives , 19 (1), 153–178. Kanefsky , B., Barlow , N. G., & Gulick, V . C. (2001). Can distributed volunteers accomplish massiv e data analysis tasks? In Pr oceedings of the 32 nd Annual Lunar and Planetary Science Conference . Houston, TX: Lunar and Planetary Institute. Kawryko w , A., Roumanis, G., Kam, A., Kwak, D., Leung, C., W u, C., et al. (2012). Phylo: A citizen science approach for improving multiple sequence alignment. PLoS One , 7 (3), e31362. Keel, W ., Chojnowski, S. D., Bennert, V . N., Schawinski, K., Lintott, C. J., Lynn, S., et al. (2012). The Galaxy Zoo survey for giant AGN-ionized clouds: P ast and present black hole accretion events. Monthly Notices of the Royal Astr onomical Society , 420 (1), 878–900. Keel, W ., Manning, A. M., Holwerda, B. W ., Mezzoprete, M., Lintott, C. J., Schawinski, K., et al. (2013). Galaxy Zoo: A catalog of overlapping galaxy pairs for dust studies. The Astronomical Society of the P acific , 125 (923), 2–16. Ke ynes, J. M. (1921). A tr eatise on pr obability . London: Macmillan. Khatib, F ., Cooper, S., T yka, M. D., Xu, K., Makedon, I., Baker , D., et al. (2011a). Algorithm discov ery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America , 108 (47), 18949–18953. Khatib, F ., DiMaio, F ., Foldit Contenders Group, Foldit V oid Crushers Group, Cooper, S., Kazmierczyk, M., et al. (2011b). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Natur e Structural and Molecular Biology , 18 (10), 1175–1177. 123 Synthese Kim, J. S., Greene, M. J., Zlateski, A., Lee, K., Richardson, M., T uraga, S. C., et al. (2014). Space-time wiring specificity supports direction selectivity in the retina. Nature , 509 (7500), 331–336. Kuhn, T . (1962). The structur e of scientific r evolutions . Chicago: Uni versity of Chicago Press. Land, K., Slosar , A., Lintott, C. J., Andreescu, D., Bamford, S., Murray , P ., et al. (2008). Galaxy Zoo: The large-scale spin statistics of spiral galaxies in the Sloan Digital Sky Survey . Monthly Notices of the Royal Astr onomical Society , 388 (4), 1686–1692. Latour , B. (2005). Reassembling the social: An introduction to actor-network-theory . New Y ork: Oxford Univ ersity Press. Leydesdorf f, L. (2001). The challenge of scientometrics . Leiden: DSWO Press. Lintott, C. J., Schawinski, K., Bamford, S., Slosar , A., Land, K., Thomas, D., et al. (2011). Galaxy Zoo 1: Data release of morphological classifications for nearly 900,000 galaxies. Monthly Notices of the Royal Astr onomical Society , 410 (1), 166–178. Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., et al. (2008). Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey . Monthly Notices of the Royal Astr onomical Society , 389 (3), 1179–1189. Lintott, C. J., Schawinski, K., Keel, W ., van Arkel, H., Bennert, N., Edmondson, E., et al. (2009). Galaxy Zoo: ‘Hanny’ s V oorwerp’, a quasar light echo? Monthly Notices of the Astronomical Society , 399 (1), 129–140. Longino, H. (1990). Science as social knowledge . Princeton, NJ: Princeton University Press. Longino, H. (2001). The fate of knowledge . Princeton, NJ: Princeton University Press. Longino, H. (2013). Studying human behavior: How scientists investigate aggr ession and sexuality . Chicago: Univ ersity of Chicago Press. McLaughlin, A. (1970). Rationality and total evidence. Philosophy of Science , 37 (2), 271–278. Méndez, B. J. H. (2008). SpaceScience@Home: Authentic research projects that use citizen scientists. In C. Garmany , M. G. Gibbs, & J. W . Moody (Eds.), EPO and a changing world: Cr eating linkages and expanding partnerships, ASP Confernce Series (V ol. 389, pp. 219–226). San Francisco: ASP Press. Nair , P . B., & Abraham, R. G. (2010). A catalog of detailed visual morphological classifications for 14,034 galaxies in the Sloan Digital Sky Surve y . Astr ophysical Journal Supplement Series , 186 (2), 427–456. Nielsen, M. (2011). Rein venting discovery: The new era of networked science . Princeton, NJ: Princeton Univ ersity Press. Nov , O., Arazy , O., Anderson, D., & (2011). Dusting for science: Motivation and participation of digital citizen science volunteers. iConference,. (2011). Proceedings (pp. 68–74). New Y ork: ACM. Page, S. (2007). The differ ence: How the power of diversity creates better gr oups, firms, schools, and societies . Princeton, NJ: Princeton Univ ersity Press. Ponciano, L., Brasileiro, F ., Simpson, R., & Smith, A. (2014). V olunteers’ engagement in human computa- tion for astronomy projects. Computing in Science & Engineering , 16 (6), 52–59. Popper , K. (1959). The logic of scientific discovery . London: Hutchinson. Popescu, B., Hanson, M. M., & Elmegreen, B. G. (2012). Age and mass for 920 large megallanic cloud clusters deriv ed from 100 million Monte Carlo simulations. The Astrophysical Journal , 751 (2), 122– 136. Price, D Jd. (1963). Little science, big science . New Y ork: Columbia Uni versity Press. Raddick, M. J., Bracey , G., Gay , P . L., Lintott, C. J., Cardamone, C., Murray , P ., et al. (2013). Galaxy Zoo: Motiv ations of citizen scientists. arXiv preprint arXiv:1303.6886 . Rotman, D., Preece, J., Hammock, J., Procita, K., Hansen, D., Parr , C., et al. (2012). Dynamic changes in motivation in collaborativ e citizen-science projects. Pr oceedings of the ACM 2012 Conference on Computer Supported Cooperative W ork (pp. 217–226). New Y ork: A CM. San Roman, I., Sarajedini, A., & Aparicio, A. (2010). Photometric properties of the M33 star cluster system. The Astr ophysical Journal , 720 (2), 1674–1683. SciStarter . (2015). Project finder . Retrieved from http:// scistarter .com/ finder/ all . Schawinski, K., Thomas, D., Sarzi, M., Maraston, C., Kaviraj, S., Joo, S., et al. (2007). Observational evidence for A GN feedback in early-type galaxies. Monthly Notices of the Royal Astr onomical Society , 382 (4), 1415–1431. Searle, J. (1990). Collective intentions and action. In P . Cohen, J. Morgan, & M. Pollack (Eds.), Intentions in communication (pp. 401–415). Cambridge: MIT Press. Shamir , L., Y erby , C., Simpson, R., von Benda-Beckmann, A. M., T yack, P ., Samarra, F ., et al. (2014). Classification of large acoustic datasets using machine learning and crowdsourcing: Application to whale calls. Acoustical Society of America , 135 (2), 953–962. 123 Synthese Silverto wn, J. (2009). A new dawn for citizen science. T r ends in Ecology & Evolution , 24 (9), 467–471. Simon, J. (2010). Knowing together: A social epistemology for socio-technical epistemic systems (Unpub- lished doctoral dissertation) . Vienna: Universität Wien. Simpson, R., Page, K., & De Roure, D. (2014). Zooniv erse: Observing the W orld’ s Largest Citizen Science Platform. Pr oceedings of the 2 nd International W eb Observatory W orkshop (pp. 1049–1054). New Yo r k : A C M . Simpson, R., Povich, M. S., K endrew , S., Lintott, C. J., Bressert, E., Arvidsson, K., et al. (2012). The milky way project first data release: A bubblier galactic disc. Monthly Notices of the Royal Astronomical Society , 424 (4), 2442–2460. Simpson, E., Roberts, S., Psorakis, I., & Smith, A. (2012). Dynamic bayesian combination of multiple i m p e r f e c tc l a s s i fi e r s .I nT .V .G u y ,M .K á r n ý ,&D .H .W o l p e r t( E d s . ) , Decision making and imperfection (pp. 1–35). Berlin: Springer . Smith, A. M., L ynn, S., Sulliv an, M., Lintott, C. J., Nugent, P . E., Botyanszki, J., et al. (2011). Galaxy zoo supernov ae. Monthly Notices of the Royal Astr onomical Society , 412 (2), 1309–1319. Swanson, A., Kosmala, M., Lintott, C., Simpson, R., Smith, A., & Packer , C. (2015). Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna. Sci- entific Data , 2 , 150026. W illett, K. W ., Lintott, C. J., Bamford, S., Masters, K., Simmons, B. D., Casteels, K. R. V ., et al. (2013). Galaxy Zoo 2: Detailed morphological classifications for 304,122 galaxies from the Sloan Digital Sky Survey . Monthly Notices of the Royal Astr onomical Society , 435 (4), 2835–2860. W inther , R. G. (2011). Part-whole science. Synthese , 178 (3), 397–427. W ittgenstein, L. (1953). Philosophical inv estigations. In R. Rhees, G. E. M. Anscombe, & G. E. M. Anscombe (Eds.), T rans . Oxford: Blackwell. W ooldridge, M. (2002). An intr oduction to multiagent systems . London: Wile y . Zollman, K. J. S. (2007). Network epistemology (Unpublished doctoral dissertation) . Irvine: University of California. Zooniv erse (2015). Publications . Retriev ed from https:// www .zooniverse.or g/ about/ publications . 123