Science and Technology Advance through Surprise

Feng Shi, University of North Carolina at Chapel Hill, Knowledge Lab fbillshi@gmail.com

James Evans, Knowledge Lab, University of Chicago, Santa Fe Institute jevans@uchicago.edu

Breakthrough discoveries and inventions involve unexpected combinations of ​contents​ including problems, methods, and natural entities, and also diverse ​contexts​ such as journals, subfields, and conferences. Drawing on data from tens of millions of research papers, patents, and researchers, we construct models that predict next year’s content and context combinations with an AUC of 95% based on embeddings constructed from high-dimensional stochastic block models, where the improbability of new combinations itself predicts up to 50% of the likelihood that they will gain outsized citations and major awards. Most of these breakthroughs occur when problems in one field are unexpectedly solved by researchers from a distant other. These findings demonstrate the critical role of surprise in advance, and enable evaluation of scientific institutions ranging from education and peer review to awards in supporting it.

19​th​ Century philosopher and scientist Charles Sanders Peirce argued that neither the logics of deduction nor induction alone could characterize the reasoning behind path-breaking new hypotheses in science, but rather their collision through a process he termed abduction. Abduction begins as expectations born of theory or tradition become disrupted by unexpected observations or findings ​(​1​)​. Surprise stimulates scientists to forge new claims that make the surprising unsurprising. Here we empirically demonstrate across the life sciences, physical sciences and patented inventions that, following Peirce, surprising hypotheses, findings and insights are the best available predictor of outsized success. But neither Peirce nor anyone since has specified where the stuff of new hypotheses came from. One account is serendipity or making the most of surprising encounters ​(​2​, ​3​)​, encapsulated in Pasteur’s oft-quoted maxim “chance favors only the prepared mind” ​(​4​)​, but this poses a paradox. The successful scientific mind must simultaneously know enough within a scientific or technological context to be surprised, and enough outside to imagine why it should not be surprised. Here we show how surprising successes systematically emerge across, rather than within researchers; most commonly when those in one field surprisingly publish problem-solving results to audiences in a distant other. This contrasts with research that focuses on inter- and multi-disciplinarity as primary sources of advance ​(​5​–​7​)​. We suggest how predictability and surprise in science and technology provide us with new tools to evaluate how scientific institutions ranging from awards and graduate education to peer review facilitate advance.

In order to identify the sources of scientific and technological surprise, we must first identify what is expected with precision. Here we follow others in modeling discovery and invention as combinatorial processes linking previous ideas, phenomena and technologies ​(​8​–​12​)​. We separate combinations of scientific contents and contexts in order to refine our expectations about normal scientific and technological developments in the future ​(​13​)​. A new scientific or technological configuration of contents—phenomena, concepts, and methods—may surprise because it has never succeeded before, despite having been considered and attempted. A new configuration of contents that cuts across divergent contexts—journals and conferences—may surprise because it has never been imagined. The separate consideration of contents and contexts allows us to contrast scientific discovery with technological search: Fields and their boundaries are clear and ever-present for scientists at all phases of scientific production, publishing and promotion, but largely invisible for technological invention and its certification in legally protected patents.

Virtually all empirical research examining combinatorial discovery and invention has deconstructed new products into collections of pairwise combinations ​(​11​)​, resting on mature analysis tools for simple graphs that define links between entity pairs. Recent research, however, has demonstrated the critical importance of higher-order structure in understanding complex networks, ranging from the hub structure of global transportation networks to clustering in neuronal networks ​(​14​)​ to stabilizing interaction between species (​15​, ​16​)​. Here we develop a method to model the frontiers of science and technology as a complex hypergraph drawn from an embedding of contents and contexts ​(​17​)​ using mixed-membership, high-dimensional stochastic block models, where each discovery or invention can be rendered as a complete set of scientif

…(Full text truncated)…