Trustworthy AI Software Engineers

With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we re-examine what it means for an AI agent to be considered a software engineer and then critically think about what makes such an agent trustworthy. \textit{Grounded} in established definitions of software engineering (SE) and informed by recent research on agentic AI systems, we conceptualise AI software engineers as participants in human-AI SE teams composed of human software engineers and AI models and tools, and we distinguish trustworthiness as a key property of these systems and actors rather than a subjective human attitude. Based on historical perspectives and emerging visions, we identify key dimensions that contribute to the trustworthiness of AI software engineers, spanning technical quality, transparency and accountability, epistemic humility, and societal and ethical alignment. We further discuss how trustworthiness can be evaluated and demonstrated, highlighting a fundamental trust measurement gap: not everything that matters for trust can be easily measured. Finally, we outline implications for the design, evaluation, and governance of AI SE systems, advocating for an ethics-by-design approach to enable appropriate trust in future human-AI SE teams.

💡 Research Summary

The paper “Trustworthy AI Software Engineers” tackles the emerging reality that large‑language‑model (LLM) based coding agents are no longer just auxiliary tools but are being positioned as members of software‑engineering (SE) teams. The authors begin by revisiting classic SE definitions from IEEE, ACM, and the Software Engineering Body of Knowledge, emphasizing that professional engineering involves a full lifecycle—requirements elicitation, design, implementation, testing, deployment, maintenance, and evolution—most of which are socio‑technical activities. They argue that an AI agent can only be called an “AI software engineer” if it can operate beyond isolated code generation and engage in the broader set of tasks that human engineers perform.

From this premise they derive six defining characteristics for AI software engineers: (1) ability to handle non‑coding SE tasks; (2) agency expressed through multi‑step planning and tool use; (3) collaborative behavior with humans and other agents; (4) respect for human values, constraints, and ethical responsibilities; (5) reasoning over repository‑level and process‑level context; and (6) acceptance of natural‑language and artifact‑based task specifications. They formalize a working definition that sets a higher bar than “autonomous coding” and aligns with the CRAFT values (comprehensive, responsible, adaptive, foundational, translational) proposed for agentic SE.

The core contribution of the paper is a multidimensional model of trustworthiness—distinct from the human attitude of trust. Trustworthiness is treated as a property of the AI system that justifies reliance, and it is dynamic, context‑dependent, and emergent from a configuration of several dimensions. The authors group these dimensions into three broad categories:

1. Technical Quality – correctness, reliability, performance, cost, maintainability, robustness, and reproducibility. These are the traditional metrics but are interpreted in the SE workflow context (e.g., a correct but extremely slow model is still untrustworthy at scale).

2. Transparency and Accountability – transparency, explainability, traceability, and accountability. The paper stresses that developers need to understand why an AI made a decision, be able to trace outputs back to inputs and models, and hold the system accountable to different stakeholder audiences.

3. Epistemic and Ethical Alignment – epistemic humility (explicit communication of uncertainty and limitations), fairness, bias mitigation, privacy, regulatory compliance, and the use of representative data. These address the limits of AI knowledge and its societal impact.

Beyond individual agents, the authors extend trustworthiness to collective practice: collaborative reflexivity, cultural sensitivity, sustainability, and broader societal impact. Negative externalities—such as reinforcing harmful narratives or marginalising groups—can erode trust even when technical metrics look good.

A significant insight is the identification of a trust measurement gap: many trustworthiness aspects are not easily quantifiable and may only surface through prolonged human‑AI interaction. Consequently, the paper calls for continuous, context‑sensitive assessment frameworks, ethics‑by‑design governance structures, and empirical studies that capture longitudinal trust dynamics in real SE settings.

In conclusion, the paper provides a roadmap for moving from productivity‑centric narratives (AI as a replacement) toward a principled view where AI software engineers are trustworthy partners. It outlines concrete requirements for AI agents, a comprehensive set of trustworthiness dimensions, and highlights the need for new evaluation methods and governance to ensure that AI‑human SE teams can operate safely, responsibly, and effectively in the coming AI‑augmented software development era.