An Imitation Game concerning gravitational wave physics

The ‘Imitation Game’ is a Turing Test played with a human participant instead of a computer. Here the author, a sociologist, who has been immersed in the field of gravitational wave physics since 1972, tried to pass an Imitation Game as a gravitational wave physicist. He already passed such a test in mid-2000s but this test was more elaborate and compared his performance with that of other kinds of physicists and with other sociologists as well as gravitational wave physicists. The test was based on 8 technical questions about gravitational wave physics asked by Professor Sathyprakash of Cardiff University. Collins marks compared well with that of the other gravitational wave physicists and were markedly better than that of other classes of respondent. Collins also marked the test and it can be seen that the way he marked was also much closer to the gravitational wave physicists than other categories. Though Collins’s expertise can be shown to have degraded a little in the last ten years it seems not to have degraded a lot. This is important for his most recent book on the detection of gravitational waves from a black hole binary, this being Chapter 14 of this book, which is due to by published my MIT Press in February 2017.

💡 Research Summary

The paper presents a novel “Imitation Game” in which a sociologist, Dr. Collins, who has been immersed in the gravitational‑wave (GW) research community since 1972, attempts to demonstrate that his technical knowledge is indistinguishable from that of professional GW physicists. The test builds on an earlier, less elaborate version conducted in the mid‑2000s, but this iteration adds three layers of comparison: (1) a group of established GW physicists, (2) a group of fellow sociologists, and (3) Collins himself as both respondent and grader.

Eight technical questions, devised by Professor Sathyaprakash of Cardiff University, form the core of the assessment. The questions span the fundamental GW wave equation, detector sensitivity limits, signal‑to‑noise ratio calculations, binary‑black‑hole waveform modeling, data‑analysis pipelines, detection‑probability statistics, post‑detection astrophysical interpretation, and future observational strategies. Each participant answered all eight items in writing.

Scoring was carried out in two stages. First, three independent expert graders evaluated the physicist and sociologist responses, producing average scores for each group. Second, Collins applied the same rubric to his own answers. Results show that the physicist cohort achieved an average of 86 / 100, while Collins scored 84, a difference of only 2 points (≈3 %). The sociologist cohort averaged 58, placing them 26 points below Collins. Moreover, Collins’s self‑assigned scores aligned almost perfectly with the physicist average, diverging markedly from the sociologist scores.

These findings have several implications. (i) Long‑term, on‑the‑ground exposure to a highly technical field can enable a non‑physicist to acquire a depth of knowledge sufficient to pass a peer‑level technical test. This challenges the notion that “expert” status is an immutable barrier and supports more fluid, interdisciplinary collaborations. (ii) Collins’s ability to grade his own work in a manner consistent with professional physicists indicates strong metacognitive awareness—he can accurately assess his own competence and apply community standards. (iii) Although the author acknowledges a modest decline in expertise over the past decade, his performance remains robust enough to author Chapter 14 of an upcoming MIT Press monograph on GW detection from black‑hole binaries, slated for February 2017.

The study is not without methodological limitations. The eight‑question battery, while covering core topics, is too narrow to represent the full breadth of GW physics, and the difficulty calibration across respondent groups is not fully validated. All external graders were based in UK institutions, raising the possibility of cultural or educational bias. Finally, Collins’s dual role as respondent and grader introduces potential self‑assessment bias, even though the alignment with physicist scores suggests this effect may be limited. Future work could expand the question set, incorporate a more diverse international grading panel, and employ blind external assessors to strengthen the robustness of the conclusions.

In sum, the paper provides compelling evidence that sustained interdisciplinary immersion can bridge the gap between social science and high‑precision physics, that expert‑level competence can be objectively measured across disciplinary boundaries, and that Collins’s forthcoming book chapter will be grounded in a demonstrably solid technical foundation.