A Methodology for assessing Agile Software Development Approaches

Agile methods provide an organization or a team the flexibility to adopt a selected subset of principles and practices based on their culture, their values, and the types of systems that they develop. More specifically, every organization or team implements a customized agile method, tailored to better accommodate its needs. However, the extent to which a customized method supports the organizational objectives, or rather the ‘goodness’ of that method is questionable. Existing agile assessment approaches focus on a comparative analysis, or are limited in scope and application. In this research, we propose a structured, systematic and comprehensive approach to assess the ‘goodness’ of agile methods. We examine an agile method based on (1) its adequacy, (2) the capability of the organization to support the adopted principles and practices specified by the method, and (3) the method’s effectiveness. We propose the Objectives, Principles and Practices (OPP) Framework to guide our assessment. The Framework identifies (1) objectives of the agile philosophy, (2) principles that support the objectives, (3) practices that are reflective of the principles, (4) the linkages between the objectives, principles and practices, and (5) indicators for each practice to assess the effectiveness of the practice and the extent to which the organization supports its implementation. In this document, we discuss our solution approach, preliminary results, and future work.

💡 Research Summary

The paper addresses the problem of evaluating how well a customized agile method aligns with an organization’s objectives, culture, and the nature of the systems it develops. Existing agile assessment approaches either compare methods against each other or focus narrowly on a limited set of practices, often ignoring the broader organizational context. To fill this gap, the authors propose the Objectives‑Principles‑Practices (OPP) framework, a structured, systematic, and comprehensive model for assessing the “goodness” of agile methods from three perspectives: adequacy, capability, and effectiveness.

The OPP framework decomposes any agile method into three hierarchical layers. Objectives capture the high‑level goals that agile seeks to achieve (e.g., rapid value delivery, customer collaboration, continuous improvement). Principles are the underlying beliefs that support those objectives (e.g., embracing change, empowering teams, fostering feedback loops). Practices are concrete techniques that embody the principles (e.g., sprint planning, pair programming, test‑driven development). The framework explicitly defines linkages between objectives and principles, and between principles and practices. Each linkage is assigned a weight reflecting its relative contribution to the higher‑level goal. This weighted‑linkage model enables quantitative scoring of how well a method’s components cover the intended objectives.

The assessment process proceeds in three stages.

1. Adequacy Assessment – Starting from the set of objectives claimed by the method, the evaluator traverses the OPP hierarchy top‑down, checking whether appropriate principles are articulated and whether those principles are reflected in concrete practices. Weighted linkage counts are summed to produce an adequacy score for each objective and principle. The authors illustrate this step with case studies of Feature‑Driven Development (FDD), eXtreme Programming (XP), and a synthetic “Method A”.

2. Capability Assessment – This stage examines whether the organization possesses the necessary environment to support the identified practices. The authors define observable properties across four dimensions: people (skills, mindset), process (workflows, standards), project (scope, risk management), and product (quality, deployment mechanisms). For each practice, a set of indicators (practice‑property pairs) is specified. Data are gathered through surveys, interviews, and document analysis, then mapped to quantitative scores that reflect the organization’s readiness.

3. Effectiveness Assessment – Here the focus shifts to the impact of the practices on achieving the original objectives. Using a bottom‑up traversal, the evaluator starts from the observed indicators, moves upward through practices and principles, and finally infers the degree to which each objective has been realized. The authors demonstrate how, for example, high scores on automated‑testing indicators support the “continuous feedback” principle, which in turn contributes to the “high‑quality software” objective.

The OPP framework distinguishes itself from prior models such as the Sidky Agile Measurement Index (SAMI) or the Agile Adoption and Improvement Model (AAIM) by avoiding a prescriptive, level‑based set of practices. Instead, it allows organizations to select objectives that matter to them, then validates whether the chosen principles and practices are sufficient and supported. The weighted‑linkage approach also provides a nuanced view of adequacy, rather than a binary “present/absent” assessment.

Findings from the pilot studies reveal nuanced insights. FDD shows a relatively balanced coverage of objectives, principles, and practices, but scores low on the “customer collaboration” objective due to missing supporting practices. XP exhibits strong practice coverage (e.g., pair programming, continuous integration) yet suffers in capability because the organization’s documentation and governance processes are weak. Method A, though hypothetical, confirms that the OPP process can be applied consistently across different contexts.

The paper’s contributions are threefold: (1) the OPP model that formalizes agile methods as an objective‑principle‑practice hierarchy, (2) a weighted‑linkage and indicator‑based assessment methodology that integrates adequacy, capability, and effectiveness, and (3) empirical validation through multiple case studies. Limitations include a reliance on expert‑derived weights and a current set of indicators that does not yet cover all possible agile practices. Future work aims to expand the indicator repository, employ machine‑learning techniques to infer weights from historical project data, and conduct longitudinal studies to validate the predictive power of the framework.

Overall, the OPP framework offers a promising avenue for organizations seeking a rigorous, context‑sensitive evaluation of their agile adoption, moving beyond superficial checklists toward a deeper understanding of how well their agile practices truly serve strategic goals.