Retrieval-Reasoning Large Language Model-based Synthetic Clinical Trial Generation

Machine learning (ML) holds great promise for clinical applications but is often hindered by limited access to high-quality data due to privacy concerns, high costs, and long timelines associated with clinical trials. While large language models (LLMs) have demonstrated strong performance in general-purpose generation tasks, their application to synthesizing realistic clinical trials remains underexplored. In this work, we propose a novel Retrieval-Reasoning framework that leverages few-shot prompting with LLMs to generate synthetic clinical trial reports annotated with binary success/failure outcomes. Our approach integrates a retrieval module to ground the generation on relevant trial data and a reasoning module to ensure domain-consistent justifications. Experiments conducted on real clinical trials from the ClinicalTrials.gov database demonstrate that the generated synthetic trials effectively augment real datasets. Fine-tuning a BioBERT classifier on synthetic data, real data, or their combination shows that hybrid fine-tuning leads to improved performance on clinical trial outcome prediction tasks. Our results suggest that LLM-based synthetic data can serve as a powerful tool for privacy-preserving data augmentation in clinical research. The code is available at https://github.com/XuZR3x/Retrieval_Reasoning_Clinical_Trial_Generation.

💡 Research Summary

The paper addresses the chronic shortage of high‑quality clinical trial data, which hampers the development of machine‑learning models for tasks such as outcome prediction, patient stratification, and eligibility screening. To alleviate this bottleneck, the authors propose a Retrieval‑Reasoning framework that leverages large language models (LLMs) in a few‑shot in‑context learning setting to generate synthetic clinical trial reports annotated with binary success/failure outcomes.

The pipeline consists of three modules. First, a retrieval module filters the entire ClinicalTrials.gov repository using drug names from DrugBank, selecting interventions that have at least three successful and three failed trials in the labeled subset. For a chosen drug and outcome label, three matching trials are sampled to serve as exemplars. Second, a reasoning module prompts the LLM (ChatGPT‑4o‑mini) to produce five concise, medically plausible reasons why trials of that drug would succeed or fail, based on the retrieved exemplars. These reasons provide domain‑consistent guidance that helps the model stay factually grounded. Third, the generation module combines the exemplar trials, the five reasons, a persona instruction (“act as a medical expert”), and a strict formatting constraint that mimics the XML‑like structure of real ClinicalTrials.gov entries. The LLM is then asked to write a new trial report with the specified outcome, while a final diversity prompt encourages novelty. Using temperature = 1.0, the authors generated 3,358 synthetic trial reports, each containing the intervention name, study design, eligibility criteria, and outcome label.

To evaluate the utility of the synthetic data, the authors fine‑tuned a BioBERT classifier under three data regimes: Synthetic‑Only, Real‑Only (using only real trials whose interventions appear in the synthetic set), and Hybrid (both synthetic and real). They also performed a ratio experiment, fixing the training size at 3,358 samples and varying the synthetic‑real proportion from 100 % synthetic to 100 % real in 20 % increments. Performance was measured on three test scenarios: (1) in‑distribution (interventions seen in synthetic data), (2) ratio (same test set as in‑distribution but with varying training mixes), and (3) out‑of‑distribution generalization (interventions never seen in synthetic data).

Results show that Hybrid fine‑tuning consistently outperforms the other two regimes. In the in‑distribution setting, Hybrid achieves 0.642 accuracy and 0.728 PR‑AUC, surpassing Synthetic‑Only (≈0.527 accuracy) and Real‑Only (≈0.545 accuracy). The ratio experiment reveals a sweet spot at 60 % synthetic + 40 % real, which yields the highest scores, indicating that synthetic data can effectively augment limited real data while preserving essential signal. In the out‑of‑distribution test, Hybrid still leads with 0.725 accuracy and 0.694 PR‑AUC, demonstrating improved generalization to unseen interventions.

The authors further analyze representation similarity using t‑SNE visualizations and cosine similarity metrics. Synthetic trials occupy regions adjacent to real trials in the embedding space, expanding the overall coverage without introducing large distributional shifts. This suggests that the synthetic data enriches the feature space, helping the classifier learn more robust decision boundaries.

Despite these promising findings, the study has limitations. The LLM‑generated reasons and narratives, while plausible, are not medically verified; expert review would be required before clinical deployment. The binary outcome label oversimplifies the nuanced results of real trials (e.g., partial efficacy, safety endpoints). The framework currently handles single‑drug interventions; extending it to combination therapies, multi‑arm studies, or longitudinal outcomes remains an open challenge. Moreover, reliance on a proprietary LLM API raises reproducibility and cost concerns.

Future work could explore multi‑label generation, incorporate additional trial attributes (e.g., adverse events, duration), develop automated validation metrics for LLM‑generated rationales, and test the approach with other LLMs or open‑source models to improve transparency. Integrating multimodal data (e.g., imaging, genomic profiles) could further enhance synthetic trial realism.

In summary, the Retrieval‑Reasoning LLM framework successfully creates high‑fidelity synthetic clinical trial reports that, when combined with real data, substantially boost BioBERT’s outcome prediction performance across both in‑distribution and out‑of‑distribution settings. This work demonstrates a viable, privacy‑preserving data augmentation strategy for clinical research and opens avenues for broader application of LLM‑driven synthetic data generation in healthcare.