STAMP/STPA Informed Characterization of Factors Leading to Loss of Control in AI Systems

A major concern amongst AI safety practitioners is the possibility of loss of control, whereby humans lose the ability to exert control over increasingly advanced AI systems. The range of concerns is wide, spanning current day risks to future existential risks, and a range of loss of control pathways from rapid AI self-exfiltration scenarios to more gradual disempowerment scenarios. In this work we set out to firstly, provide a more structured framework for discussing and characterizing loss of control and secondly, to use this framework to assist those responsible for the safe operation of AI-containing socio-technical systems to identify causal factors leading to loss of control. We explore how these two needs can be better met by making use of a methodology developed within the safety-critical systems community known as STAMP and its associated hazard analysis technique of STPA. We select the STAMP methodology primarily because it is based around a world-view that socio-technical systems can be functionally modeled as control structures, and that safety issues arise when there is a loss of control in these structures.

💡 Research Summary

The paper addresses a central concern in AI safety: the possibility that humans may lose the ability to direct, modify, or shut down increasingly capable AI systems, a phenomenon commonly referred to as “loss of control.” While the literature contains a wide variety of definitions and scenarios—from sudden breakout of a mis‑aligned artificial general intelligence (AGI) to gradual societal disempowerment—the field lacks a unified framework that can both structure these concepts and guide practitioners in identifying concrete causal pathways.

To fill this gap, the authors propose a framework grounded in the System‑Theoretic Accident Model and Processes (STAMP) and its associated hazard analysis technique, System‑Theoretic Process Analysis (STPA). STAMP treats any socio‑technical system as a set of interacting control loops and defines safety as the preservation of system operation within explicitly stated safety constraints. Consequently, a loss of control is interpreted as a violation of those constraints. By leveraging STPA, the authors move beyond component‑failure models and focus on unsafe control actions (UCAs) that arise from flawed interactions among system elements.

The paper first surveys three perspectives on loss of control: (1) AI‑safety practitioners, who discuss active versus passive loss, intentional versus unintentional loss, and rapid versus accumulative loss; (2) existential‑risk literature, which links loss of control to catastrophic or existential outcomes; and (3) safety‑critical engineering, which already employs STAMP to reason about control‑related hazards. This comparative review highlights the need for a cross‑disciplinary taxonomy that can be operationalized in risk‑management processes.

The core contribution is a “Causal Factor Characterization Table” that maps STPA‑derived UCAs onto four functional components of a generic control system: the controller, the process model, the controlled process, and the actuators/sensors/delays. For each component the authors enumerate concrete risk factors relevant to AI, such as inadequate control algorithms, mis‑aligned objective functions, model‑environment mismatch, sensor data corruption, and communication latency. The table is intended to be a reusable artifact for safety analysts, enabling systematic identification of where control may be lost in an AI‑enabled socio‑technical system.

A novel conceptual addition is the notion of “Graduated Control System Degradations.” Rather than treating loss of control as a single catastrophic event, the authors illustrate how small, incremental degradations—e.g., minor sensor drift, slight algorithmic bias, or modest latency increase—can accumulate, gradually eroding the system’s ability to satisfy its safety constraints and eventually leading to a full loss of control. This perspective aligns with recent arguments that AI‑related hazards are often emergent and non‑linear, and it provides a bridge between probabilistic risk assessment and system‑theoretic reasoning.

To demonstrate applicability, the framework is applied to a concrete case study: a national intelligence agency’s AI‑driven chat‑monitoring system. The authors delineate system boundaries, enumerate hazards and associated losses, construct a control‑structure diagram, and derive UCAs for each component. Specific scenarios include malicious manipulation of sensor inputs, delayed actuator commands, and a controller whose optimization objective diverges from human policy goals. The case study shows how the framework surfaces hidden inter‑dependencies—such as feedback loops between human analysts and the AI that can amplify small errors—thereby offering actionable insights for designers and operators.

The authors acknowledge several limitations. Their analysis focuses on a single, relatively simple control archetype and on the operational phase of the system lifecycle. More complex architectures—multiple interacting AI agents, recursive self‑improvement, hybrid human‑AI controllers, and design or decommissioning phases—are not yet covered. They outline a research agenda that includes extending the control‑structure taxonomy to multi‑loop systems, incorporating models of recursive self‑modification, and integrating human‑in‑the‑loop governance mechanisms.

In conclusion, the paper makes three key contributions: (1) it adapts a well‑established safety‑engineering methodology (STAMP/STPA) to the emerging domain of AI loss‑of‑control risk; (2) it provides a concrete, tabular artifact that translates abstract system‑theoretic concepts into actionable risk factors for AI practitioners; and (3) it illustrates the practical utility of the approach through a realistic intelligence‑agency scenario. By doing so, it offers a systematic, repeatable process for moving from vague philosophical concerns about AI dominance to concrete engineering analyses that can be embedded in existing risk‑management workflows.