Exploring the Impact of Dataset Statistical Effect Size on Model Performance and Data Sample Size Sufficiency

Having a sufficient quantity of quality data is a critical enabler of training effective machine learning models. Being able to effectively determine the adequacy of a dataset prior to training and evaluating a model’s performance would be an essential tool for anyone engaged in experimental design or data collection. However, despite the need for it, the ability to prospectively assess data sufficiency remains an elusive capability. We report here on two experiments undertaken in an attempt to better ascertain whether or not basic descriptive statistical measures can be indicative of how effective a dataset will be at training a resulting model. Leveraging the effect size of our features, this work first explores whether or not a correlation exists between effect size, and resulting model performance (theorizing that the magnitude of the distinction between classes could correlate to a classifier’s resulting success). We then explore whether or not the magnitude of the effect size will impact the rate of convergence of our learning rate, (theorizing again that a greater effect size may indicate that the model will converge more rapidly, and with a smaller sample size needed). Our results appear to indicate that this is not an effective heuristic for determining adequate sample size or projecting model performance, and therefore that additional work is still needed to better prospectively assess adequacy of data.

💡 Research Summary

**
The paper tackles a long‑standing practical problem in machine learning: how to assess, before training, whether a given dataset is large and informative enough to yield a well‑performing model. The authors propose using simple descriptive statistics—namely, the effect size of each feature with respect to the target label—as a prospective heuristic. For continuous variables they compute Cohen’s d, for categorical variables they compute odds ratios, and then they average these values across all features to obtain a single “average effect size” for a dataset (or a subset of it).

To test the usefulness of this metric, the authors conduct two families of experiments on the UCI Adult census dataset (≈48 k rows, 14 features). After standard preprocessing (removing missing rows, label‑encoding categorical variables, standardizing numerics) they retain about 33 k samples. They then create 66 disjoint subsets, each containing 500 randomly selected rows, and treat each subset as an independent “dataset”. For each subset they calculate the average effect size and train four classic classifiers—logistic regression, decision tree, random forest, and a shallow neural network—on the same feature set. Model performance is evaluated using accuracy, precision, recall, and F1‑score.

Experiment 1 examines the direct correlation between average effect size and the four performance metrics. Pearson and Spearman correlation coefficients are computed across the 66 points. The results show very weak linear and monotonic relationships (|r|≈0.1–0.2) with p‑values far above conventional significance thresholds, indicating that effect size does not reliably predict performance. A second variant of Experiment 1 swaps the label (using gender instead of income) and repeats the procedure; the same lack of correlation is observed, suggesting that the finding is not label‑specific.

Experiment 1‑2 explores whether removing a single feature with a known effect size leads to a predictable drop in performance. For each of the 14 features the authors drop it from the full dataset, recompute the effect size of the omitted feature, retrain the four classifiers, and measure the performance degradation. The scatter plot of “effect size of removed feature” versus “performance drop” again shows no systematic pattern; some high‑effect‑size features cause only modest drops, while some low‑effect‑size features cause larger declines.

Experiment 2 shifts focus from final performance to learning dynamics. For each model trained on each subset, learning curves (error versus training‑set size) are fitted with a logarithmic function, and the slope (or the rate at which validation error converges) is extracted. The authors then correlate these slopes with the corresponding average effect sizes. The analysis yields negligible correlations, and a similar result is obtained when examining the rate at which the gap between training and validation error shrinks as more data are added. Thus, a larger effect size does not appear to accelerate convergence or reduce the amount of data needed for a model to “bottom out”.

Overall, the empirical evidence presented suggests that a simple average of univariate effect sizes is not an effective proxy for data sufficiency in supervised classification tasks. The authors discuss several reasons for this outcome. First, averaging effect sizes discards information about feature interactions, multicollinearity, and the high‑dimensional geometry that modern classifiers exploit. Second, effect size is a univariate measure that captures only the mean difference (or odds) between two classes; it does not reflect distributional overlap, variance heterogeneity, or non‑linear separability. Third, the study is limited to a single dataset and relatively small subsets (500 samples), which may not capture the diversity of real‑world data regimes.

The paper concludes by recommending future work that goes beyond univariate effect size. Potential directions include multivariate distance metrics (e.g., Mahalanobis distance), information‑theoretic measures such as mutual information, Bayesian sample‑size planning, or simulation‑based learning‑curve extrapolation. Moreover, extending the analysis to deep neural networks, larger and more heterogeneous datasets, and incorporating class imbalance would provide a more comprehensive picture of how statistical properties of data relate to model learning. In sum, while the idea of a pre‑training data‑quality metric is appealing, the current study demonstrates that simple effect‑size heuristics are insufficient for reliably predicting model performance or required sample size.