SETA: Statistical Fault Attribution for Compound AI Systems

Modern AI systems increasingly comprise multiple interconnected neural networks to tackle complex inference tasks. Testing such systems for robustness and safety entails significant challenges. Current state-of-the-art robustness testing techniques, whether black-box or white-box, have been proposed and implemented for single-network models and do not scale well to multi-network pipelines. We propose a modular robustness testing framework that applies a given set of perturbations to test data. Our testing framework supports (1) a component-wise system analysis to isolate errors and (2) reasoning about error propagation across the neural network modules. The testing framework is architecture and modality agnostic and can be applied across domains. We apply the framework to a real-world autonomous rail inspection system composed of multiple deep networks and successfully demonstrate how our approach enables fine-grained robustness analysis beyond conventional end-to-end metrics.

💡 Research Summary

The paper addresses a pressing challenge in modern AI: the increasing prevalence of compound systems composed of multiple, specialized neural networks that together perform complex inference tasks. While modular architectures simplify development and improve performance, they also introduce new safety concerns because faults in upstream components can cascade through the pipeline, leading to catastrophic failures. Existing robustness testing methods—whether black‑box end‑to‑end (E2E) attacks, neuron‑coverage guided fuzzing, or formal verification tools such as Reluplex, ERAN, or Marabou—are designed for single‑model analysis and cannot pinpoint which sub‑module is responsible for a failure, nor can they capture emergent errors that arise from interactions among modules.

To fill this gap, the authors propose SETA (Statistical Fault Attribution), a modular testing framework that combines Metamorphic Testing (MT) with Execution Trace Analysis. The core idea is to define oracle‑free specifications for each component using Metamorphic Relations (MRs). An MR expresses a required relationship between the output of a component on an original input x and on a transformed input ˜x = g(x). The paper provides concrete MR families for four common vision tasks: image classification (label invariance or bounded L∞‑norm deviation), object localization (IoU ≥ τ), object detection (combined label and IoU constraints for each detected object), and semantic segmentation (pixel‑wise equality or relaxed IoU per class). MRs are expressed uniformly using Kronecker‑δ (equality) and Heaviside‑H (inequality) functions, allowing both discrete labels and continuous scores to be handled within a single mathematical framework.

Beyond defining MRs, SETA records the dynamic execution graph generated when a perturbed test sample traverses the system. Each node corresponds to a component, and the framework annotates the node with a binary flag indicating whether its MR(s) hold. By scanning the graph, SETA automatically identifies the earliest node where a violation occurs—the “fault origin”—and tracks how that violation propagates downstream. To accommodate components that require multiple checks, the authors introduce Composite Metamorphic Relations (CMRs): the logical AND of all individual MRs for a component yields a component‑level score S_i ∈ {0,1}. The system‑wide score S = ∏_i S_i is 1 only if every component satisfies all its MRs, providing a clear, hierarchical indicator of overall consistency.

The framework is deliberately architecture‑ and modality‑agnostic. It can be plugged into existing MLOps pipelines (Kedro for data pipelines, MLflow for experiment tracking, Evidently AI for drift monitoring) without replacing them; instead, SETA adds a diagnostic layer that supplies fine‑grained fault localization. Users can extend the MR library by adding new relations as plugins, making the approach adaptable to domains such as medical imaging, autonomous driving, or industrial inspection.

The authors validate SETA on a real‑world autonomous rail‑inspection system that chains three deep networks: (1) an image classifier, (2) an object detector, and (3) a segmentation model. They generate a suite of realistic perturbations (synthetic fog, rain, motion blur, additive noise) and compare traditional E2E robustness metrics (overall accuracy drop) with SETA’s component‑wise analysis. While E2E metrics merely show a modest degradation, SETA reveals that many failures originate in the object detection module where IoU falls below the prescribed threshold; this, in turn, causes the segmentation model to mislabel large regions. In other cases, the classifier remains robust, but small coordinate errors in detection accumulate and trigger downstream segmentation faults. These findings illustrate how SETA can expose hidden vulnerabilities that would otherwise remain undetected.

Key contributions highlighted are: (1) a unified framework that couples metamorphic testing with execution tracing for fault attribution; (2) oracle‑free, scalable MR definitions that work for black‑box components; (3) a compositional scoring system that aggregates component‑level checks into a system‑wide consistency metric; and (4) empirical evidence that SETA can pinpoint failure origins in a multi‑stage AI pipeline where existing verification or monitoring tools fall short.

The paper also discusses limitations. Generating diverse perturbations and constructing execution traces incurs computational overhead, which may be prohibitive for real‑time systems. Moreover, the binary nature of MR satisfaction may overlook nuanced degradations that are still acceptable in practice. Future work is suggested in the direction of adaptive perturbation generation (e.g., reinforcement‑learning‑guided attacks) and lightweight tracing mechanisms to reduce runtime cost.

In summary, SETA offers a practical, extensible solution for robustness testing of compound AI systems. By providing fine‑grained, causally grounded fault attribution, it enables developers and safety engineers to identify, prioritize, and remediate weaknesses in complex pipelines—an essential step toward trustworthy AI in safety‑critical applications such as autonomous transportation, healthcare diagnostics, and infrastructure inspection.