As Artificial Intelligence (AI) systems increasingly assume consequential decision-making roles, a widening gap has emerged between technical capabilities and institutional accountability. Ethical guidance alone is insufficient to counter this challenge; it demands architectures that embed governance into the execution fabric of the ecosystem. This paper presents the Ten Criteria for Trustworthy Orchestration AI, a comprehensive assurance framework that integrates human input, semantic coherence, audit and provenance integrity into a unified Control-Panel architecture. Unlike conventional agentic AI initiatives that primarily focus on AI-to-AI coordination, the proposed framework provides an umbrella of governance to the entire AI components, their consumers and human participants. By taking aspiration from international standards and Australia's National Framework for AI Assurance initiative, this work demonstrates that trustworthiness can be systematically incorporated (by engineering) into AI systems, ensuring the execution fabric remains verifiable, transparent, reproducible and under meaningful human control.
The advent of increasingly autonomous AI systems has fundamentally shifted the central question of AI research. Where earlier generations asked, "can machines think?", the contemporary challenge have become, "can machines by governed, trusted and aligned with human values?". This transformation reflects a growing recognition that technical capabilities of a system alone are insuOicient. AI systems operating in sensitive environments must demonstrate verifiable trustworthiness throughout their operational lifecycle.
Current LLM-based systems exhibit persistent challenges of trustworthiness that undermine institutional confidence. Lack of transparency in decision-making processes prevents meaningful oversight, while model drift introduces unpredictable behavioural changes over time. Moreover, non-reproducibility of outputs complicates verification and audit; consequently, leaving stakeholders unable to comprehend, contest, or correct system behaviours. These limitations represent fundamental barriers to responsible deployment in domains where errors carry substantial weight, negatively impacting public safety and financial stability. Furthermore, as agentic AI approaches promise enhanced capabilities through AI-to-AI coordination, they typically focus inward on interagent functionality without adequate attention to the governance structures necessary for safe downstream consumption.
As a result of above-mentioned challenges, we have AI deployments with widen accountability gap leading to a divergence between what AI systems can do and what institutions can verify, explain and control. This gap cannot be bridged through ethical guidelines or post-hoc auditing alone. It requires architectural solutions that embed governance directly into the computational fabric of the system; thus, presenting an opportunity to contribute. The proposed Orchestration AI Framework presented in this paper is response to this opportunity.
The Orchestration AI Framework represents a distinct paradigm that extends beyond conventional agentic approaches by integrating humans, AI modules, and information systems into a unified, continuously supervised ecosystem. Through a central Control-Plane, the framework maintains persistent oversight of all interactions; thus, enforcing policies, validating semantic exchanges, and anchoring provenance records in real time.
This paper presents the Ten Criteria for Trustworthy Orchestration AI, a comprehensive assurance framework that operationalises trustworthiness as a set of interdependent, verifiable system components. Specifically, this research aims to: (1) articulate a coherent set of criteria that collectively define Trustworthy AI Orchestration; (2) demonstrate how these criteria can be implemented as runtime properties rather than external controls; and (3) provide a foundation for compliance assessment aligned with international standards and national AI governance frameworks.
The governance of artificial intelligence has attracted substantial attention from standards bodies, regulatory authorities, and research communities worldwide. While substantial attention has been directed toward AI trustworthiness by standards bodies and regulatory authorities, significant gaps remain regarding the application of these principles to orchestrated, multi-component systems.
At the international level, the ISO/IEC standards family establishes the baseline for organisational accountability. The most relevant standard ISO/IEC 38507:2022 (International Organization for Standardization [ISO] provides essential guidance for governing bodies on enabling organisational AI use, establishing higher-level accountability and governance principles for decision-making, data usage, and risk management. As part of the ISO/IEC 38500 series, it extends established IT governance frameworks to address AI-specific considerations including ethical use, compliance, and stakeholder expectations. However, the standard operates at the management level without prescribing architectural properties that AI systems must exhibit, particularly for orchestrated, multi-component ecosystems where governance must be enforced as a runtime property rather than an oversight.
Similarly, International Electrotechnical Commission ([IEC], 2022) and ISO/IEC 42001:2023 provide essential guidance on governance implications and management systems, emphasising board-level responsibility and risk assessment. On the other hand, ISO/IEC 23894:2023 complements these structured approaches for identifying and treating AI-related risks. However, while these standards establish critical organisational procedures, they operate primarily at the management system level and do not prescribe the specific architectural properties required for technical implementation.
On the other side, ISO group has also understood the technical aspects of AI implementations; thus, complementary standards with technical perspective have emerged. For example, the ISO/IE
This content is AI-processed based on open access ArXiv data.