VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMs

Large language models (LLMs) are increasingly being used to generate synthetic datasets for the evaluation and training of downstream models. However, prior work has noted that such generated data lacks diversity. In this paper, we propose Voyager, a novel principled approach to generate diverse datasets. Our approach is iterative and directly optimizes a mathematical quantity that optimizes the diversity of the dataset using the machinery of determinantal point processes. Furthermore, our approach is training-free, applicable to closed-source models, and scalable. In addition to providing theoretical justification for the working of our method, we also demonstrate through comprehensive experiments that Voyager significantly outperforms popular baseline approaches by providing a 1.5-3x improvement in diversity.

💡 Research Summary

The rapid advancement of Large Language Models (LLMs) has positioned synthetic data generation as a cornerstone for training and evaluating downstream AI models. However, a critical bottleneck persists: the inherent lack of diversity in LLM-generated outputs, often characterized by repetitive patterns and mode collapse. This paper introduces “Voyager,” a groundbreaking, training-free approach designed to systematically maximize the diversity of synthetic datasets.

The core innovation of Voyager lies in its integration of Determinantal Point Processes (DPP) into an iterative generation framework. Unlike traditional methods that rely on complex fine-tuning or intricate prompt engineering—which often fail to escape the model’s high-probability, repetitive output zones—Voyager treats dataset construction as a subset selection problem. By leveraging the mathematical properties of DPP, the framework can model “repulsion” between data points. In essence, it measures the similarity between potential data candidates and selects a subset that maximizes the determinant of a kernel matrix, effectively maximizing the volume of the feature space spanned by the selected samples.

A significant advantage of Voyager is its “training-free” nature. Because the optimization occurs during the selection and iterative refinement phase rather than through weight updates, the method is fully compatible with closed-source, API-based models like GPT-4. This makes the approach highly accessible, cost-effective, and scalable to massive datasets. The authors provide rigorous theoretical justifications for why this mathematical approach leads to superior diversity and demonstrate its efficacy through extensive experimentation.

The empirical results are highly impressive, showing that Voyager achieves a 1.5x to 3x improvement in diversity metrics compared to existing state-of-the-art baselines. By providing a principled way to generate high-entropy, non-redundant datasets, Voyager offers a scalable solution to the “diversity crisis” in synthetic data. This technology has profound implications for the future of AI development, particularly in domains where edge-case coverage and high-fidelity, diverse data are crucial for building robust and unbiased models, such as autonomous driving, medical diagnostics, and complex reasoning tasks. Voyager represents a paradigm shift from simple generative prompting to mathematically optimized data curation.