A recipe for irreproducible results

Recent studies have shown that many results published in peer-reviewed scientific journals are not reproducible. This raises the following question: why is it so easy to fool myself into believing that a result is reliable when in fact it is not? Using Brownian motion as a toy model, we show how this can happen if ergodicity is assumed where it is unwarranted. A measured value can appear stable when judged over time, although it is not stable across the ensemble: a different result will be obtained each time the experiment is run.

💡 Research Summary

The paper addresses the reproducibility crisis by exposing a subtle statistical pitfall: the unwarranted assumption of ergodicity when analysing noisy data. Using Brownian motion—a paradigmatic stochastic process—as a toy model, the authors demonstrate that a measurement protocol that works perfectly for ergodic systems can systematically produce apparently stable but fundamentally irreproducible results when applied to a non‑ergodic (specifically weakly non‑ergodic) system.

The authors first outline a generic experimental strategy. Step 1 designs a protocol that extracts a “true value” from a noisy observable by computing a finite‑time average and stopping the measurement once the observable’s rate of change falls below a preset threshold ε. In an ergodic system, the time average converges to the ensemble expectation, guaranteeing that repeated experiments yield the same result regardless of the unknown initial state.

The paper then reviews ergodicity concepts. In a fully ergodic dynamical system, any trajectory eventually samples the entire phase space, so the long‑time average equals the invariant measure’s expectation. Strong ergodicity breaking (disconnected basins) trivially destroys reproducibility. The authors focus on weak ergodicity breaking, where the ensemble distribution is stationary but individual trajectories do not converge to a single value.

Brownian motion (W(t)) is introduced as a universal example of weak ergodicity breaking. Although the ensemble mean of (W(t)) is zero, the finite‑time average (W_t = \frac{1}{t}\int_0^t W(s)ds) has a variance that grows as (t^3/3). Paradoxically, the derivative of this time average, (dW_t/dt), has a mean‑square that decays as (1/(3t)). Consequently, the “time variation” metric \