Synthetic Data, Information, and Prior Knowledge: Why Synthetic Data Augmentation to Boost Sample Doesn't Work for Statistical Inference

The use of synthetic data to deidentify data and to improve predictive models is well-attested to. The augmentation of datasets using synthetically generated data is an alluring proposition: in the best case, it generates realistic data \textit{in silico} at a fraction of the cost of authentic data which may be found \textit{in vivo} or \textit{in vitro}. This poses novel epistemic challenges. We contend that synthetic data augmentation is best understood as a novel way of accounting for prior knowledge. In this manuscript, we propose a definition of synthetic distributions and analyze how synthetic data augmentation interplays with standard accounts of maximum likelihood and Bayesian estimation. We observe that the marginal Fisher information contributed by synthetic data processes is subject to fundamental bounds, and enumerate obstacles to the use of synthetic data augmentation to aid in inferential tasks. We then articulate a Bayesian formulation of the way that synthetic data augmentation can be coherently understood, but argue that naive approaches to the specification of the prior are epistemically unjustifiable. This suggests that enhanced scrutiny must be placed on identifying justifiable priors to warrant the use and inclusion of data drawn from specific synthetic distributions. While our analysis shows the challenges and limitations of using synthetic data augmentation to improve upon traditional statistical model reasoning, it does suggest that augmentation is the principal approach analysts using outcome reasoning (i.e. using train/test splits to justify the analysis) can constrain an otherwise high-dimensional model space, providing an alternative to trying to encode the constraints into the potentially complex architecture of the algorithm.

💡 Research Summary

The manuscript tackles the increasingly popular practice of augmenting real datasets with synthetically generated observations, a technique that promises cost‑effective data expansion and privacy‑preserving data sharing. The authors argue that synthetic data augmentation should be viewed primarily as a mechanism for encoding prior knowledge rather than as a source of new information about the underlying population parameters.

To formalize this view, they define a “synthetic distribution” S as a mapping from an empirical sample X (drawn from a space Ω) to a probability distribution over a possibly different space Ξ. This abstract definition encompasses classic non‑parametric bootstrapping, weighted bootstraps, class‑conditional resampling, and even transformations that enforce geometric symmetries (e.g., rotations of images). The key point is that synthetic data need not be a realistic replica of the true data‑generating process; it merely reflects a user‑specified prior or structural constraint.

The core technical contribution consists of two information‑theoretic results. Theorem 5 shows that when a synthetic sample S is generated conditional on the observed real sample X, the conditional Fisher information I_{S|X}(θ) is zero. Consequently, the joint Fisher information of (X, S) equals the Fisher information of X alone, I_{X,S}(θ)=I_X(θ). In other words, synthetic observations add no marginal information about the true parameter θ beyond what is already contained in the original data. Theorem 8 further establishes an upper bound: the Fisher information contained in the synthetic sample alone, I_S(θ), cannot exceed I_X(θ). This rules out any “free lunch” in inference—synthetic data cannot magically increase the precision of maximum‑likelihood estimators or shrink standard errors beyond the limits imposed by the original sample size.

From a Bayesian perspective, the authors treat a synthetic distribution as a posterior distribution P(·|X) derived from a prior P and the real data X. By invoking the Bayesian reflection principle, they argue that sampling from this posterior yields no additional information beyond what is already encoded in the posterior itself. Hence, augmenting a dataset with draws from its own posterior does not improve posterior inference; any benefit must come from an external, well‑justified prior that captures genuine domain knowledge.

The paper surveys three broad use‑cases. First, data masking (e.g., for privacy‑preserving release of census data) is a legitimate application because the synthetic data are explicitly tied to the original sample, and standard inferential procedures remain valid. Second, human‑in‑the‑loop augmentation (e.g., chat‑bot feedback) and third, physics‑informed synthetic data for MRI reconstruction illustrate how prior knowledge can be injected without redesigning model architectures. In these scenarios, the synthetic data serve as a conduit for prior constraints rather than as a source of new statistical evidence.

The authors caution against indiscriminate use of synthetic data to inflate sample size for routine regression or classification tasks. Because the added observations do not increase Fisher information, any apparent performance gains are likely due to reduced variance from regularization or model selection effects, not from genuine information gain. Moreover, successful use of synthetic data requires transparent knowledge of (a) the original sample size and composition used to train the synthetic generator, and (b) the exact dependence structure between the synthetic and real data. Without this transparency, analysts cannot correctly assess the information content of the augmented dataset.

In conclusion, synthetic data are valuable as a flexible tool for encoding domain‑specific priors, especially when model architecture constraints are undesirable. However, for statistical inference—whether frequentist maximum‑likelihood or Bayesian posterior analysis—synthetic augmentation does not provide additional Fisher information and must be justified at the level of prior specification. Practitioners should therefore treat synthetic data augmentation as a modeling choice that encodes prior beliefs, not as a shortcut to improve estimator efficiency or confidence interval precision.