An Empirical Study on Decision making for Quality Requirements

[Context] Quality requirements are important for product success yet often handled poorly. The problems with scope decision lead to delayed handling and an unbalanced scope. [Objective] This study characterizes the scope decision process to understand influencing factors and properties affecting the scope decision of quality requirements. [Method] We studied one company’s scope decision process over a period of five years. We analyzed the decisions artifacts and interviewed experienced engineers involved in the scope decision process. [Results] Features addressing quality aspects explicitly are a minor part (4.41%) of all features handled. The phase of the product line seems to influence the prevalence and acceptance rate of quality features. Lastly, relying on external stakeholders and upfront analysis seems to lead to long lead-times and an insufficient quality requirements scope. [Conclusions] There is a need to make quality mode explicit in the scope decision process. We propose a scope decision process at a strategic level and a tactical level. The former to address long-term planning and the latter to cater for a speedy process. Furthermore, we believe it is key to balance the stakeholder input with feedback from usage and market in a more direct way than through a long plan-driven process.

💡 Research Summary

The paper investigates how quality requirements (QRs), also known as non‑functional requirements, are handled during scope‑decision making in a large B2C software‑intensive product line. Over a five‑year period (2010‑2015) the authors examined the company’s scope‑decision database, which contained 4,446 feature entries across more than 40 products and 36 releases, and conducted semi‑structured interviews with senior engineers who participated in the decision process. Their analysis identified 196 quality‑focused features (QFs), representing only 4.41 % of all features. These QFs were classified into performance, security, user‑experience, and power‑efficiency categories.

A key finding is that the prevalence and acceptance rate of QFs vary markedly with the product‑line lifecycle stage. During the early “build‑up” phase (2010‑2012) the company’s strategic focus was technology‑driven, emphasizing new hardware and core software capabilities; consequently, few QFs were proposed and those that were submitted had low acceptance. In the subsequent “growth‑to‑maturity” phase (2013‑2015) market competition intensified, and the organization began to value customer‑centric attributes. This shift led to a noticeable increase in both the number of QF proposals and their approval rates, indicating that quality concerns become more prominent when the product strategy moves from pure innovation to market fit.

The study also reveals the influence of stakeholder structures on QR handling. External stakeholders (partners, large customers) often initiated QR proposals. Their requests followed a formal “proposal → upfront analysis → approval” workflow that incurred an average lead‑time of six months or more. Such long lead‑times conflicted with release schedules, causing many quality items to be postponed or dropped. Internal stakeholders (marketing, technical experts) tended to generate QR ideas based on post‑release user data and market feedback. However, these ideas were usually addressed through ad‑hoc hot‑fixes rather than being integrated into the formal scope‑decision pipeline, limiting their visibility and systematic treatment.

Based on these observations, the authors propose a two‑level scope‑decision framework. At the strategic level, a “quality mode” is explicitly embedded in the long‑term product roadmap, defining quality goals, target levels, and the proportion of resources allocated to QR development. At the tactical level, a rapid feedback loop is introduced, leveraging analytics, performance monitoring, A/B testing, and usage telemetry to feed real‑time quality insights directly into the decision process. This dual‑level approach aims to reduce reliance on external, slow‑moving analyses, shorten lead‑times, and ensure that quality considerations are balanced with functional feature planning.

While the paper does not provide a quantitative link between QR handling and market success, it convincingly demonstrates that quality features are systematically under‑represented in scope decisions and that this under‑representation is tied to both the product‑line maturity stage and the stakeholder configuration. The authors call for future work to model the causal relationship between QR investment and product performance, and to empirically evaluate the proposed framework in other domains. Their findings contribute valuable longitudinal evidence to the requirements engineering literature, highlighting the need for explicit, data‑driven processes to manage quality requirements in complex, market‑driven product lines.