Toward Third-Party Assurance of AI Systems: Design Requirements, Prototype, and Early Testing

As Artificial Intelligence (AI) systems proliferate, the need for systematic, transparent, and actionable processes for evaluating them is growing. While many resources exist to support AI evaluation, they have several limitations. Few address both the process of designing, developing, and deploying an AI system and the outcomes it produces. Furthermore, few are end-to-end and operational, give actionable guidance, or present evidence of usability or effectiveness in practice. In this paper, we introduce a third-party AI assurance framework that addresses these gaps. We focus on third-party assurance to prevent conflict of interest and ensure credibility and accountability of the process. We begin by distinguishing assurance from audits in several key dimensions. Then, following design principles, we reflect on the shortcomings of existing resources to identify a set of design requirements for AI assurance. We then construct a prototype of an assurance process that consists of (1) a responsibility assignment matrix to determine the different levels of involvement each stakeholder has at each stage of the AI lifecycle, (2) an interview protocol for each stakeholder of an AI system, (3) a maturity matrix to assess AI systems’ adherence to best practices, and (4) a template for an assurance report that draws from more mature assurance practices in business accounting. We conduct early validation of our AI assurance framework by applying the framework to two distinct AI use cases – a business document tagging tool for downstream processing in a large private firm, and a housing resource allocation tool in a public agency – and conducting expert validation interviews. Our findings show early evidence that our AI assurance framework is sound and comprehensive, usable across different organizational contexts, and effective at identifying bespoke issues with AI systems.

💡 Research Summary

The paper addresses the growing need for systematic, transparent, and actionable evaluation of AI systems that are increasingly embedded in societal and economic processes. While a plethora of responsible AI (RAI) tools, internal governance frameworks, and audit guidelines exist, they suffer from several critical shortcomings: they often focus solely on technical outcomes, neglect the full AI lifecycle, provide abstract guidance that is difficult to operationalize, and lack empirical evidence of usability and effectiveness. Moreover, internal audits can suffer from conflicts of interest, whereas external audits may lack access to necessary data, leaving a gap for an impartial, end‑to‑end assurance mechanism.

To fill this gap, the authors propose a third‑party AI assurance framework grounded in five design requirements (R1‑R5): (R1) simultaneous focus on process and outcomes; (R2) coverage of all stakeholders and lifecycle stages; (R3) provision of clear, actionable instructions; (R4) delivery of concrete improvement recommendations; and (R5) validation through empirical usability and effectiveness studies.

The framework consists of four tightly integrated components:

1. Responsibility Assignment Matrix (RAM) – a RACI‑style matrix that maps each lifecycle phase (value proposition, data collection & preprocessing, model design & training, deployment & monitoring) to the appropriate responsible, accountable, consulted, and informed parties. This clarifies role boundaries, reduces ambiguity, and enables traceability of decisions.

2. Stakeholder Interview Protocol – a set of tailored interview guides for designers, developers, operators, end‑users, and impacted communities. The protocol probes whether responsibilities are being discharged, how risks are perceived, the transparency of decision‑making, and the extent of feedback incorporation.

3. Maturity Matrix – a rubric that defines best‑practice criteria for each lifecycle stage and rates an organization’s current practice on a five‑point scale. The matrix serves both as a diagnostic tool to assess AI governance maturity and as a roadmap for incremental improvement.

4. Assurance Report Template – modeled after financial audit reports, this template structures findings, risk assessments, actionable recommendations, and a monitoring plan, facilitating clear communication to both internal stakeholders and external regulators or the public.

The authors conduct early validation through two case studies that differ markedly in domain, scale, and stakeholder composition: (a) a private‑sector document‑tagging system used for downstream processing, and (b) a public‑sector housing resource allocation tool. Applying the framework uncovered concrete issues such as data labeling bias, insufficient model explainability, misalignment with policy objectives, and inadequate stakeholder feedback loops. The resulting assurance reports offered specific remediation steps, including revising data pipelines, instituting explainability dashboards, and establishing formal community consultation mechanisms.

Complementing the case studies, semi‑structured interviews with eight AI governance practitioners (industry experts, auditors, and policy analysts) provided external validation. Participants praised the framework’s comprehensiveness, the practicality of the RAM and interview guides, and the clarity of the report template. They also highlighted the value of third‑party involvement in mitigating conflicts of interest. Critiques focused on the need for flexibility to adapt the framework to organizations of varying size and maturity, and on guidance for optimal timing of assurance activities (e.g., early‑stage versus post‑deployment).

Overall, the paper makes several substantive contributions: it articulates a clear distinction between AI audits (outcome‑focused, punitive) and AI assurance (process‑and‑outcome‑focused, constructive), it operationalizes a set of design requirements into concrete artifacts, and it provides initial empirical evidence of the framework’s necessity, soundness, usability, and effectiveness. The authors acknowledge that their work is a prototype; future research must pursue longitudinal, large‑scale studies, develop automation support (e.g., tool‑assisted maturity scoring), and explore alignment with emerging standards such as NIST’s AI Risk Management Framework, ISO/IEC 42001, and sector‑specific regulations. By establishing a standardized, third‑party assurance methodology, the paper lays groundwork for a professionalized AI assurance ecosystem analogous to financial assurance, thereby enhancing trust, early risk detection, and compliance across the rapidly expanding AI landscape.