Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalance

Imbalanced classification and spurious correlation are common challenges in data science and machine learning. Both issues are linked to data imbalance, with certain groups of data samples significantly underrepresented, which in turn would compromise the accuracy, robustness and generalizability of the learned models. Recent advances have proposed leveraging the flexibility and generative capabilities of large language models (LLMs), typically built on transformer architectures, to generate synthetic samples and to augment the observed data. In the context of imbalanced data, LLMs are used to oversample underrepresented groups and have shown promising improvements. However, there is a clear lack of theoretical understanding of such synthetic data approaches. In this article, we develop novel theoretical foundations to systematically study the roles of synthetic samples in addressing imbalanced classification and spurious correlation. Specifically, we first explicitly quantify the benefits of synthetic oversampling. Next, we analyze the scaling dynamics in synthetic data augmentation, and derive the corresponding scaling law. Finally, we demonstrate the capacity of transformer models to generate high-quality synthetic samples. We further conduct extensive numerical experiments to validate the efficacy of the LLM-based synthetic oversampling and augmentation.

💡 Research Summary

The paper tackles the pervasive problem of class imbalance and spurious correlations that degrade the performance, robustness, and fairness of machine learning models. While traditional oversampling methods such as random replication, SMOTE, and ADASYN mitigate imbalance by synthetically increasing minority samples, they suffer from limited diversity and over‑fitting. Recent generative approaches using GANs or large language models (LLMs) have shown empirical promise, yet a rigorous statistical understanding has been missing.

The authors propose a unified theoretical framework for synthetic oversampling and augmentation powered by LLMs, specifically transformer‑based GPT‑2 (fine‑tuned) and GPT‑4 (prompt‑only). The workflow converts tabular data into natural‑language sentences, feeds them to an LLM, and then deserializes the generated text back into synthetic tabular records. Algorithm 1 outlines (1) selection of balanced seed data, (2) generation of a large pool of synthetic samples, (3) minority‑only oversampling by randomly drawing the required number of synthetic records, and (4) uniform augmentation by adding a fixed number N of synthetic records to every group.

The theoretical contributions are threefold. First, the authors define group‑specific risk (R^{(g)}(\theta)) and a balanced risk (R_{\text{bal}}(\theta)) that averages across all groups. They introduce the bias term (B^{(g)}(\theta)) measuring the discrepancy between the loss under synthetic and real distributions. Under smoothness, positive‑definite Hessian, and a vanishing bias assumption (the synthetic distribution converges to the real one as the seed size grows), they prove that synthetic oversampling reduces the excess risk for the minority group, moving the empirical minimizer toward the oracle balanced solution (\theta_{\text{bal}}).

Second, for synthetic augmentation, they derive a scaling law that quantifies how the excess balanced risk (R_{\text{bal}}(\hat\theta)-R_{\text{bal}}(\theta_{\text{bal}})) decays as a function of the total number of synthetic samples (N). The decay follows a polynomial rate (O(N^{-\gamma})), where the exponent (\gamma) depends on the average imbalance ratio (\rho) and the quality of synthetic data (the magnitude of (B^{(g)})). The law shows that higher imbalance (larger (\rho)) demands more synthetic data to achieve the same risk reduction, providing a principled guideline for data‑budget allocation.

Third, the paper establishes that transformer‑based LLMs satisfy the required bias‑vanishing condition. By leveraging the expressive power of large pre‑trained models and fine‑tuning on a modest seed set, the synthetic distribution’s total variation distance to the real distribution shrinks at a rate that fulfills Assumption 2. Empirically, GPT‑4 prompted generation yields lower bias and higher sample diversity than fine‑tuned GPT‑2, confirming the theoretical predictions.

Extensive experiments validate the theory. In binary classification tasks with severe imbalance (e.g., rare disease detection), LLM‑based oversampling outperforms SMOTE, ADASYN, and GAN‑based methods on F1‑score, AUC, and calibration metrics. For augmentation, the authors vary (N) and observe the balanced excess risk decreasing roughly as (N^{-0.5}), matching the derived scaling law. Additional ablation studies demonstrate that the choice of seed size, prompting strategy, and model size affect the bias term and consequently the risk reduction.

In summary, the paper delivers a rigorous statistical foundation for using LLMs to generate synthetic data that mitigates class imbalance and spurious correlations. It provides actionable scaling laws for practitioners to estimate how much synthetic data is needed given a specific imbalance ratio and desired performance target. By bridging theory and practice, the work opens a new avenue for reliable, model‑agnostic data augmentation in high‑stakes domains such as healthcare, finance, and security, where data imbalance is a critical bottleneck.