An Empirical Study on the Procedure to Derive Software Quality Estimation Models

Software quality assurance has been a heated topic for several decades. If factors that influence software quality can be identified, they may provide more insight for better software development management. More precise quality assurance can be achieved by employing resources according to accurate quality estimation at the early stages of a project. In this paper, a general procedure is proposed to derive software quality estimation models and various techniques are presented to accomplish the tasks in respective steps. Several statistical techniques together with machine learning method are utilized to verify the effectiveness of software metrics. Moreover, a neuro-fuzzy approach is adopted to improve the accuracy of the estimation model. This procedure is carried out based on data from the ISBSG repository to present its empirical value.

💡 Research Summary

The paper addresses the longstanding challenge of accurately estimating software quality early in a project’s life‑cycle, arguing that reliable predictions enable more effective allocation of resources and risk mitigation. After reviewing prior work—most of which suffers from ad‑hoc metric selection, limited statistical validation, and inadequate handling of non‑linear relationships—the authors propose a comprehensive, five‑step procedure for deriving software quality estimation models.

Step 1 defines the specific quality targets (e.g., defect density, mean time to repair, maintenance cost) that the model will predict. Step 2 involves data acquisition from the ISBSG repository, extracting over 2,500 project records with variables such as size (LOC, function points), development duration, team size, programming language, and process maturity (CMMI level). The raw data undergoes cleaning, missing‑value imputation, outlier removal via inter‑quartile range, and Z‑score normalization.

In Step 3, the authors perform rigorous variable selection. Pearson and Spearman correlation coefficients identify initial relationships, while variance inflation factor (VIF) analysis eliminates multicollinearity (VIF > 5). A stepwise regression then isolates statistically significant predictors; development size, team experience, and process maturity emerge as the strongest contributors to quality outcomes.

Step 4 builds predictive models. A baseline linear regression is complemented by four machine‑learning algorithms: CART decision trees, Random Forests, Support Vector Regression, and a multilayer perceptron neural network. Ten‑fold cross‑validation guards against over‑fitting, and performance is measured using RMSE, MAE, and R². Random Forest yields the highest R² (0.78) but sacrifices interpretability, whereas linear regression and CART retain clearer explanatory power.

Step 5 introduces a neuro‑fuzzy (NF) system to capture complex, non‑linear interactions among the selected metrics. Expert‑derived fuzzy rules map input variables to linguistic terms; these rules are then fine‑tuned together with neural‑network weights via back‑propagation. The NF model reduces RMSE by roughly 12 % and MAE by about 10 % relative to the best pure machine‑learning model, demonstrating a tangible accuracy gain.

Empirical validation proceeds on two fronts. Statistically, all retained predictors achieve p‑values below 0.01, confirming their relevance. Practically, the NF‑enhanced model is applied to a set of real‑world projects, where predicted defect densities deviate from observed values by an average of less than 8 %, indicating strong early‑stage forecasting capability.

The discussion highlights the procedure’s strengths: systematic metric selection, integration of statistical rigor with modern machine‑learning, and the NF layer’s ability to model non‑linearities. Limitations include reliance on a single public repository (ISBSG), which may not capture domain‑specific nuances, and the expert‑driven construction of fuzzy rules, which can be labor‑intensive. Future work is suggested in three areas: expanding the dataset to cover embedded, cloud, and mobile domains; automating fuzzy‑rule generation through data‑driven clustering; and embedding the estimation framework into real‑time project‑management tools for continuous quality monitoring.

In conclusion, the authors demonstrate that a disciplined, multi‑method approach can substantially improve software quality estimation accuracy. By coupling statistical validation, diverse machine‑learning techniques, and a neuro‑fuzzy refinement, the proposed methodology offers a practical pathway for managers to make data‑informed decisions at the earliest phases of software development, thereby enhancing overall project success.