Practitioner Insights on Fairness Requirements in the AI Development Life Cycle: An Interview Study

Nowadays, Artificial Intelligence (AI), particularly Machine Learning (ML) and Large Language Models (LLMs), is widely applied across various contexts. However, the corresponding models often operate as black boxes, leading them to unintentionally act unfairly towards different demographic groups. This has led to a growing focus on fairness in AI software recently, alongside the traditional focus on the effectiveness of AI models. Through 26 semi-structured interviews with practitioners from different application domains and with varied backgrounds across 23 countries, we conducted research on fairness requirements in AI from software engineering perspective. Our study assesses the participants’ awareness of fairness in AI / ML software and its application within the Software Development Life Cycle (SDLC), from translating fairness concerns into requirements to assessing their arising early in the SDLC. It also examines fairness through the key assessment dimensions of implementation, validation, evaluation, and how it is balanced with trade-offs involving other priorities, such as addressing all the software functionalities and meeting critical delivery deadlines. Findings of our thematic qualitative analysis show that while our participants recognize the aforementioned AI fairness dimensions, practices are inconsistent, and fairness is often deprioritized with noticeable knowledge gaps. This highlights the need for agreement with relevant stakeholders on well-defined, contextually appropriate fairness definitions, the corresponding evaluation metrics, and formalized processes to better integrate fairness into AI/ML projects.

💡 Research Summary

This paper, “Practitioner Insights on Fairness Requirements in the AI Development Life Cycle: An Interview Study,” presents a qualitative investigation into how software practitioners perceive, define, and operationalize fairness requirements within the AI/ML development lifecycle. Acknowledging the critical need for fairness alongside model effectiveness in an era of pervasive AI, the study moves beyond algorithmic definitions to explore the practical, software engineering challenges of implementing fairness.

The research is based on 26 semi-structured interviews with AI practitioners (including data scientists, ML engineers, and product managers) from diverse application domains and 23 countries. Using thematic analysis, the study addresses several key research questions centered on the Software Development Life Cycle (SDLC).

The findings reveal a significant gap between awareness and consistent practice. While practitioners broadly understand fairness as the avoidance of discrimination based on sensitive attributes (e.g., race, gender), translating this abstract concern into actionable software requirements is a major hurdle. Definitions of fairness are often vague and context-dependent, leading to confusion about which specific fairness metrics (e.g., demographic parity, equalized odds) to apply.

The study uncovers that fairness considerations are frequently inconsistent and deprioritized within the SDLC. Concerns about fairness often arise informally or late in the process, rather than being systematically elicited and documented as formal requirements during the early stages. Implementation and validation practices are ad-hoc; while some teams employ fairness testing and metrics, there is no standardized approach. A critical finding is the central role of trade-offs, where fairness is most commonly sacrificed to meet pressing project deadlines. It also competes with other priorities like model accuracy (effectiveness), security, functionality, and user experience.

The root causes of these challenges are identified as not merely technical but also organizational and human-centric. They include a lack of agreed-upon, context-specific fairness definitions, knowledge gaps among team members, insufficient stakeholder alignment (e.g., between engineers, product managers, and legal teams), and the absence of formalized processes or incentives to prioritize fairness.

In conclusion, the paper argues that achieving fairness in AI requires more than just technical debiasing tools. It necessitates: (1) early stakeholder consensus on well-defined, contextual fairness criteria, (2) the establishment of formalized processes to integrate fairness throughout the entire SDLC—from requirements elicitation to deployment, and (3) addressing organizational cultures that undervalue fairness against competing business objectives. The study provides valuable empirical evidence from the field, highlighting the urgent need for practical frameworks, guidelines, and educational resources to bridge the gap between fairness research and industry practice.