Software Process Measurement and Related Challenges in Agile Software Development: A Multiple Case Study

Existing scientific literature highlights the importance of metrics in Agile Software Development (ASD). Still, empirical investigation into metrics in ASD is scarce, particularly in identifying the rationale and the operational challenges associated with metrics. Under the Q-Rapids project (Horizon 2020), we conducted a multiple case study at four Agile companies, using the Goal Question Metric (GQM) approach, to investigate the rationale explaining the choice of process metrics in ASD, and challenges faced in operationalizing them. Results reflect that companies are interested in assessing process aspects like velocity, testing performance, and estimation accuracy, and they prefer custom metrics for these assessments. Companies use metrics as a means to access and even capitalize on the data, erstwhile inaccessible due to technical or process constraints. However, development context of a company can hinder metrics operationalization, manifesting primarily as unavailability of the data required to measure metrics…(check the paper for full Abstract)

💡 Research Summary

This paper presents a multiple‑case study investigating why software‑intensive companies choose particular process metrics in Agile Software Development (ASD) and what challenges they face when operationalising those metrics. The study was carried out under the EU Horizon 2020 Q‑Rapids project and involved four companies of varying size, domain, and agile practice (Scrum, Scrumban, and an ad‑hoc agile approach). Data were collected through twelve Goal‑Question‑Metric (GQM) workshops (three per company) with a total of 19 practitioners (project managers, product owners, quality managers, and developers).

A total of 132 distinct metrics were elicited, most of which belong to the “process factors” category. The most frequently mentioned metrics across all companies relate to (1) Velocity – measuring the number of issues completed per sprint to assess planning accuracy and identify bottlenecks; (2) Testing Performance – capturing execution time, test success density, and defect detection rates to monitor quality‑assurance efficiency; (3) Issue Estimation Accuracy – comparing estimated effort versus actual effort to improve resource planning; (4) Code Quality – tracking changes in source‑code quality indicators such as defect injection rate; and (5) Testing Status – unit‑test success density and coverage. Companies consistently preferred custom‑built metrics over off‑the‑shelf ones because they could tailor the measures to their specific business goals, product characteristics, and organisational culture.

Two major categories of operational challenges emerged. The first is data availability. While Company A (medium‑size, Scrum) and Company C (large, ad‑hoc agile) already possessed relatively integrated toolchains, Company B (large, globally distributed) suffered from heterogeneous logging formats and fragmented data sources, making consistent metric extraction difficult. Company D (small) lacked any automated data‑collection pipeline, forcing manual data extraction and increasing the risk of errors. The second challenge is uncertainty about the actionable value of metrics. Practitioners view metrics as a means to gain data visibility and to foster a data‑driven culture, yet they often cannot translate metric outcomes into concrete decision‑making (e.g., whether a drop in velocity is due to staffing shortages or technical debt). This gap reduces confidence in the metric’s usefulness and hampers its adoption.

The study highlights the strong influence of development context on metric selection and operationalisation. Company A, with a stable Scrum process, sought a standardised set of metrics to increase transparency across the organisation. Company B, with many sub‑teams, required both a common baseline and team‑specific extensions to accommodate local nuances. Company C, which follows a flexible, principle‑based agile approach, designed highly customised metrics that align with its product‑maturity goals. Company D, operating with a core team that frequently changes members, focused on a minimal set of metrics aimed at early detection of design, security, and platform issues. These differences demonstrate that a “one‑size‑fits‑all” metric framework is unrealistic; metric design must be contextualised to size, process, and tooling.

By comparing the academic literature’s typical agile metrics (e.g., burn‑down charts, cumulative flow diagrams) with the custom metrics actually used in industry, the paper reveals a gap: prior work often proposes metrics without empirically validating their feasibility in real‑world settings, especially in large, distributed organisations. The authors therefore call for future research that (i) quantifies the causal link between specific metrics and concrete improvement actions, (ii) develops automated data‑pipeline and dashboard solutions to reduce collection overhead, and (iii) conducts longitudinal studies to assess the long‑term impact of metric‑driven process improvement.

In summary, the paper provides empirical evidence that software‑intensive companies are motivated to adopt process metrics for transparency and data‑driven decision‑making, but they encounter significant hurdles related to data accessibility and the translation of metric results into actionable insights. Development context plays a decisive role in shaping both the choice of metrics and the nature of the challenges, underscoring the need for flexible, context‑aware metric frameworks in Agile environments.