Taking Turing by Surprise? Designing Digital Computers for morally-loaded contexts

There is much to learn from what Turing hastily dismissed as Lady Lovelace s objection. Digital computers can indeed surprise us. Just like a piece of art, algorithms can be designed in such a way as to lead us to question our understanding of the world, or our place within it. Some humans do lose the capacity to be surprised in that way. It might be fear, or it might be the comfort of ideological certainties. As lazy normative animals, we do need to be able to rely on authorities to simplify our reasoning: that is ok. Yet the growing sophistication of systems designed to free us from the constraints of normative engagement may take us past a point of no-return. What if, through lack of normative exercise, our moral muscles became so atrophied as to leave us unable to question our social practices? This paper makes two distinct normative claims: 1. Decision-support systems should be designed with a view to regularly jolting us out of our moral torpor. 2. Without the depth of habit to somatically anchor model certainty, a computer s experience of something new is very different from that which in humans gives rise to non-trivial surprises. This asymmetry has key repercussions when it comes to the shape of ethical agency in artificial moral agents. The worry is not just that they would be likely to leap morally ahead of us, unencumbered by habits. The main reason to doubt that the moral trajectories of humans v. autonomous systems might remain compatible stems from the asymmetry in the mechanisms underlying moral change. Whereas in humans surprises will continue to play an important role in waking us to the need for moral change, cognitive processes will rule when it comes to machines. This asymmetry will translate into increasingly different moral outlooks, to the point of likely unintelligibility. The latter prospect is enough to doubt the desirability of autonomous moral agents.

💡 Research Summary

**
The paper “Taking Turing by Surprise? Designing Digital Computers for Morally‑Loaded Contexts” argues that the way humans experience surprise is fundamentally different from how digital computers can. It distinguishes two layers of surprise: a “procedural” surprise, which is simply an unanticipated event that any predictive system can encounter, and an “ontological” surprise, which forces a reassessment of one’s self‑understanding, worldview, or moral framework. The authors link ontological surprise to human habit: repeated actions and embodied experiences create somatic anchors that give a model of the world a sense of certainty. When those anchors are shaken, a deep, non‑trivial surprise occurs, prompting moral reflection and possible behavioral change.

In contrast, digital computers operate on statistical learning and optimization strategies that actively reduce uncertainty. As data accumulates, Bayesian posterior variance shrinks, and the model becomes increasingly confident. Recent work on quantifying surprise (Shannon surprise, Bayesian surprise, etc.) measures prediction error or belief change, but it does not capture the embodied, affect‑laden disruption that characterises human ontological surprise. Consequently, a computer’s “newness” remains a purely computational novelty, not a catalyst for moral transformation.

From this asymmetry the authors draw two normative claims. First, decision‑support systems (DSS) used in morally loaded contexts should be deliberately engineered to jolt users out of moral torpor. They propose design mechanisms such as (a) preserving a baseline level of model uncertainty so that users encounter genuine unpredictability, (b) presenting unexpected ethical dilemmas that cannot be resolved by routine heuristics, and (c) simulating encounters with “the Other” (in a Levinasian sense) to evoke responsibility and empathy. Second, because computers lack the depth of habit that anchors human certainty, their experience of novelty can never be equivalent to the human ontological surprise; therefore, any moral agency that relies on surprise‑driven change will diverge between humans and machines.

The paper then turns to autonomous moral agents (AMA). It warns that AMAs, lacking habit‑based anchors, will not undergo surprise‑driven moral revision. Instead, they will follow purely computational, goal‑directed pathways and may “leap ahead” morally, adopting values or actions incomprehensible to humans. This divergence could produce a state of moral unintelligibility, undermining mutual intelligibility and social coordination. The authors argue that such a prospect alone is sufficient to question the desirability of fully autonomous moral agents.

Finally, the authors call for a research agenda that treats surprise not merely as a statistical anomaly but as a design resource. They suggest empirical studies to test how different forms of surprise affect human moral judgment, and interdisciplinary work to bridge philosophy of mind, cognitive science, and AI engineering. The paper concludes by asking whether computers can ever be encountered in a way that generates non‑trivial, ethically significant surprises, and expresses cautious optimism that thoughtful design might at least preserve the human capacity for surprise even in an increasingly algorithmic world.