A Reproducible Framework for Bias-Resistant Machine Learning on Small-Sample Neuroimaging Data

We introduce a reproducible, bias-resistant machine learning framework that integrates domain-informed feature engineering, nested cross-validation, and calibrated decision-threshold optimization for small-sample neuroimaging data. Conventional cross-validation frameworks that reuse the same folds for both model selection and performance estimation yield optimistically biased results, limiting reproducibility and generalization. Demonstrated on a high-dimensional structural MRI dataset of deep brain stimulation cognitive outcomes, the framework achieved a nested-CV balanced accuracy of 0.660,$\pm$,0.068 using a compact, interpretable subset selected via importance-guided ranking. By combining interpretability and unbiased evaluation, this work provides a generalizable computational blueprint for reliable machine learning in data-limited biomedical domains.

💡 Research Summary

The paper addresses a pervasive problem in neuroimaging machine‑learning: small sample sizes combined with extremely high‑dimensional feature spaces lead to optimistic performance estimates and poor reproducibility. The authors propose a fully reproducible, bias‑resistant framework that integrates three core components: (1) domain‑informed feature engineering that transforms raw regional volumes from structural MRI into a compact set of biologically meaningful composites, (2) strict nested cross‑validation (CV) that separates model selection from performance estimation, and (3) probability calibration together with decision‑threshold optimization to produce stable, clinically relevant operating points.

Dataset and preprocessing: The study retrospectively analyzes 332 patients who underwent pre‑surgical structural MRI for deep brain stimulation (DBS). T1‑weighted scans are processed with FreeSurfer and SynthSeg, yielding 109 regional volumes after quality control. Patients are dichotomized into “low‑risk” and “elevated‑risk” groups based on a cognitive risk score (CRS ≥ 3). No global z‑scaling is applied; instead, the authors compute volume fractions normalized by total intracranial volume (TIV), ventricle‑to‑brain ratios, gray/white‑matter ratios, deep gray‑matter aggregates, bilateral asymmetry indices, and low‑order interactions (e.g., age × ventricular fraction). This results in a feature set that preserves anatomical interpretability while reducing redundancy.

Modeling pipeline: All classifiers are implemented with scikit‑learn (v1.5) and a fixed random seed (42). The outer loop uses a 5‑fold stratified CV to obtain an unbiased estimate of generalization performance. Within each outer‑train split, a 3‑fold inner CV performs hyper‑parameter grid search, using average‑precision as the selection criterion. After the inner search, the best model is retrained on the full outer‑train data, and a sigmoid (Platt) calibrator is fitted on the same training scores. The calibrated probabilities are then used to find a single decision threshold t* that maximizes balanced accuracy (BA) on the outer‑train predictions. This threshold is applied to the untouched outer‑test fold. Crucially, hyper‑parameter tuning, calibration, and threshold search never see outer‑test data, eliminating data leakage.

Results: A wide panel of classifiers (Random Forest, Extra Trees, Gradient Boosting, k‑Nearest Neighbors, Linear Discriminant Analysis, Multilayer Perceptron, Naïve Bayes, etc.) is evaluated. On the raw SynthSeg features, Random Forest achieves a mean BA of 0.656 ± 0.056; on the engineered feature set, the same model reaches 0.660 ± 0.068, the highest reported performance. Extra Trees yields comparable results (0.696 ± 0.042 on raw vs. 0.713 ± 0.050 on engineered). The improvement of roughly +0.01 in BA across models demonstrates that the anatomically informed composites add discriminative signal without increasing model complexity. Probability calibration improves AUC from ~0.70 to ~0.72, Brier score to ~0.22, and expected calibration error (ECE) to ~0.13, indicating well‑calibrated predictions. The optimal threshold is tightly clustered around 0.39 (IQR ≈ 0.01) across all outer folds, confirming threshold stability.

Comparison with naïve evaluation: When the same data are assessed with a single train‑test split or naïve CV (where the same folds are reused for hyper‑parameter tuning and performance estimation), balanced accuracy drops to the 0.57–0.59 range and variability increases. This stark contrast underscores how the nested design eliminates the ~20 % optimistic bias reported in prior neuroimaging studies.

Feature importance analysis: Averaged Random Forest importances reveal that ventricle‑to‑brain ratio, specific cortical region fractions (e.g., right paracentral, left superior frontal), and asymmetry measures rank highest, providing clinically interpretable markers that could guide future DBS candidate selection.

Reproducibility: The entire workflow—including preprocessing, feature engineering, nested CV, calibration, and threshold optimization—is scripted with fixed library versions and random seeds. The authors commit to releasing the code and configuration files upon publication, ensuring that other researchers can exactly replicate the results from a single configuration file.

Conclusion: By unifying domain‑driven feature construction, leakage‑free nested cross‑validation, and calibrated decision‑threshold selection, the authors deliver a generalizable, reproducible blueprint for machine‑learning on small‑sample, high‑dimensional biomedical data. The framework not only yields unbiased performance estimates (balanced accuracy ≈ 0.66) but also produces interpretable models suitable for clinical deployment, addressing a critical gap in neuroimaging predictive analytics.