Stable but Wrong: When More Data Degrades Scientific Conclusions

Modern science increasingly relies on ever-growing observational datasets and automated inference pipelines, under the implicit belief that accumulating more data makes scientific conclusions more reliable. Here we show that this belief can fail in a fundamental and irreversible way. We identify a structural regime in which standard inference procedures converge smoothly, remain well calibrated, and pass conventional diagnostic checks, yet systematically converge to incorrect conclusions. This failure arises when the reliability of observations degrades in a manner that is intrinsically unobservable to the inference process itself. Using minimal synthetic experiments, we demonstrate that in this regime additional data do not correct error but instead amplify it, while residual-based and goodness-of-fit diagnostics remain misleadingly normal. These results reveal an intrinsic limit of data-driven science: stability, convergence, and confidence are not sufficient indicators of epistemic validity. We argue that inference cannot be treated as an unconditional consequence of data availability, but must instead be governed by explicit constraints on the integrity of the observational process.

💡 Research Summary

The paper challenges the widely held assumption that accumulating more observational data inevitably leads to more reliable scientific conclusions. The authors identify a structural regime—termed “unobservable reliability drift”—in which standard statistical inference procedures appear well‑behaved (stable, convergent, well‑calibrated) yet systematically converge to incorrect parameter values.

Theoretical framework: data are generated as yₜ = θ* + εₜ + bₜ, where εₜ is ordinary observation noise and bₜ is a slowly varying bias that cannot be distinguished from noise within any finite observation window. Conventional estimators that assume bₜ ≡ 0 remain consistent under the usual stationarity and identifiability assumptions, but when those assumptions fail, the estimator ˆθₙ converges almost surely to θ* + limₙ→∞ (1/n)∑ₜbₜ. If the time‑averaged drift is non‑zero, the estimator converges stably to a biased value, and this bias persists regardless of how much data are collected because it stems from structural non‑identifiability rather than sampling variability.

Synthetic experiments: The authors construct minimal synthetic scenarios to isolate the phenomenon. In a linear drift case, posterior means converge smoothly while posterior variances contract, giving the illusion of increasing confidence. Residuals retain near‑zero mean and appropriate variance, so standard residual diagnostics (e.g., χ² tests) show no alarm. A random‑walk drift (non‑linear) reproduces the same pattern, demonstrating robustness to the drift’s functional form. Sliding‑window “forgetting” strategies fail because the drift evolves more slowly than the window length, leaving the bias intact.

Paradox of more data: Crucially, the absolute inference error grows monotonically with the number of observations when drift is present, as shown in Figure 3. In a no‑drift control (Figure 4), error declines as expected. This counter‑intuitive result arises because each new observation reinforces the posterior’s belief in the biased estimate, tightening the confidence interval around the wrong value and thereby amplifying the systematic error.

Real‑world validation: To demonstrate that the effect is not merely theoretical, the authors analyze the Sloan Digital Sky Survey (SDSS) Stripe 82 stellar photometry. The mean stellar color ⟨g − r⟩ should be stationary, yet over a decade‑long baseline it exhibits a statistically significant monotonic drift (Kendall’s τ > 0, p ≪ 0.01). As more observations are added, the estimated mean’s uncertainty shrinks, but the estimate converges to a biased limit rather than the true stationary value. No anomalous residuals or calibration warnings appear, mirroring the synthetic findings.

Implications: The phenomenon is not a failure of model complexity, optimization, or algorithmic design; it is a fundamental epistemic limitation. When observational integrity degrades in ways that are invisible to the inference pipeline, traditional signals of reliability—stability, convergence, confidence—become misleading. Consequently, scientific practice must treat inference as a governed activity, requiring external validation of the data acquisition process (e.g., independent calibration checks, ground‑truth benchmarks, or procedural audits).

In summary, the paper provides both a rigorous theoretical proof and empirical evidence that more data can, under certain hidden drift conditions, degrade rather than improve scientific conclusions. It calls for a paradigm shift: data abundance alone is insufficient; explicit, externally verified constraints on observational integrity are essential to safeguard epistemic validity.