The Necessity of a Holistic Safety Evaluation Framework for AI-Based Automation Features

The intersection of Safety of Intended Functionality (SOTIF) and Functional Safety (FuSa) analysis of driving automation features has traditionally excluded Quality Management (QM) components from rigorous safety impact evaluations. While QM components are not typically classified as safety-relevant, recent developments in artificial intelligence (AI) integration reveal that such components can contribute to SOTIF-related hazardous risks. Compliance with emerging AI safety standards, such as ISO/PAS 8800, necessitates re-evaluating safety considerations for these components. This paper examines the necessity of conducting holistic safety analysis and risk assessment on AI components, emphasizing their potential to introduce hazards with the capacity to violate risk acceptance criteria when deployed in safety-critical driving systems, particularly in perception algorithms. Using case studies, we demonstrate how deficiencies in AI-driven perception systems can emerge even in QM-classified components, leading to unintended functional behaviors with critical safety implications. By bridging theoretical analysis with practical examples, this paper argues for the adoption of comprehensive FuSa, SOTIF, and AI standards-driven methodologies to identify and mitigate risks in AI components. The findings demonstrate the importance of revising existing safety frameworks to address the evolving challenges posed by AI, ensuring comprehensive safety assurance across all component classifications spanning multiple safety standards.

💡 Research Summary

The paper addresses a critical gap in current automotive safety engineering: the treatment of artificial‑intelligence (AI) components that are classified as Quality‑Management (QM) items under ISO 26262 functional safety (FuSa) but can still generate Safety‑of‑Intended‑Functionality (SOTIF) hazards. Traditional FuSa focuses on systematic and random hardware/software failures and assigns Automotive Safety Integrity Levels (ASIL) based on severity, exposure, and controllability. Components that do not reach an ASIL threshold are labeled QM and are typically excluded from rigorous safety analysis. In contrast, SOTIF, governed by ISO 21448, deals with hazards arising from functional insufficiencies, foreseeable misuse, and operating‑design‑domain (ODD) violations—issues that are especially relevant to AI‑driven perception and decision modules.

The authors first argue theoretically that AI models are inherently probabilistic and data‑dependent; therefore, even a perfectly functioning QM‑rated module can produce unsafe outputs when confronted with edge‑case inputs, biased training data, or environmental conditions outside its training distribution. They illustrate this with a Level 2+ advanced driver‑assistance system (ADAS) where a low‑level perception (LLP) block, implemented with convolutional neural networks (CNNs), is QM‑rated. Nevertheless, under low‑light, glare, or adverse weather, the CNN’s object‑detection accuracy can drop dramatically, leading to incorrect inputs for higher‑level perception (HLP) and downstream control functions such as Adaptive Cruise Control, Lane‑Keeping Assist, or Automated Emergency Braking. Because no hardware fault occurs, FuSa would not flag the situation, yet SOTIF criteria (C > 0 or S > 0) deem it a residual risk that must be addressed.

To systematically capture these risks, the paper adopts System‑Theoretic Process Analysis (STPA). STPA models the entire socio‑technical control structure—including sensors, AI accelerators, data pipelines, human operators, and the environment—to identify Unsafe Control Actions (UCAs). The authors map specific UCAs to concrete ADAS scenarios (e.g., missed pedestrian detection in rain, lane‑line mis‑recognition on a construction site) and demonstrate how a QM‑rated AI component can be the root cause of a hazardous control action.

The core contribution is a proposed integrated safety‑assessment framework that merges FuSa, SOTIF, and the emerging AI safety standard ISO/PAS 8800. ISO/PAS 8800 introduces a lifecycle‑oriented set of requirements for AI: data governance, model design and verification, continuous monitoring, and risk management. The authors show how each of these activities can be embedded into the traditional FuSa development process (e.g., extending the safety plan to include data‑set diversity analysis, adding model‑uncertainty quantification to the verification phase, and implementing runtime anomaly detection that triggers a safe‑stop maneuver). By doing so, the framework provides both qualitative (e.g., ALARP, PRB) and quantitative (e.g., statistical confidence in simulation‑based validation) acceptance criteria for SOTIF hazards, even when the underlying component remains QM‑rated.

Risk‑mitigation measures recommended include: (1) expanding training data to cover the full ODD, using synthetic data generation for rare scenarios; (2) employing uncertainty‑aware inference (e.g., Bayesian neural networks) to flag low‑confidence predictions; (3) introducing redundancy through multi‑sensor fusion to reduce reliance on a single AI block; and (4) establishing a runtime safety monitor that can override or disengage the AI‑driven function when safety thresholds are breached.

The paper’s findings underscore that excluding QM‑rated AI modules from safety analysis is no longer defensible as autonomous driving systems become more capable and complex. A holistic approach that treats AI‑specific functional insufficiencies as first‑class safety concerns is essential for meeting regulatory expectations and ensuring that the net safety benefit of higher‑level automation outweighs any newly introduced risks. The authors conclude by calling for industry‑wide adoption of the integrated framework and for future research on standardized metrics for AI‑related SOTIF risk quantification.