The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data

Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities. By transforming data from a scarce resource into a controllable asset, LLMs mitigate the bottlenecks imposed by the acquisition costs of real-world data for model training, evaluation, and system iteration. However, ensuring the high quality of LLM-generated synthetic data remains a critical challenge. Existing research primarily focuses on generation methodologies, with limited direct attention to the quality of the resulting data. Furthermore, most studies are restricted to single modalities, lacking a unified perspective across different data types. To bridge this gap, we propose the \textbf{LLM Data Auditor framework}. In this framework, we first describe how LLMs are utilized to generate data across six distinct modalities. More importantly, we systematically categorize intrinsic metrics for evaluating synthetic data from two dimensions: quality and trustworthiness. This approach shifts the focus from extrinsic evaluation, which relies on downstream task performance, to the inherent properties of the data itself. Using this evaluation system, we analyze the experimental evaluations of representative generation methods for each modality and identify substantial deficiencies in current evaluation practices. Based on these findings, we offer concrete recommendations for the community to improve the evaluation of data generation. Finally, the framework outlines methodologies for the practical application of synthetic data across different modalities.

💡 Research Summary

**
The paper introduces the “LLM Data Auditor” framework, a systematic approach for evaluating synthetic data generated by large language models (LLMs) across six major modalities: text, symbolic/logical reasoning, tabular, semi‑structured (graph, JSON, logs), vision‑language, and agent data. The authors argue that current research focuses heavily on generation techniques and extrinsic evaluation—measuring downstream model performance—while largely ignoring intrinsic properties of the data itself, especially aspects of trustworthiness such as privacy, safety, fairness, and robustness.

The framework is organized into four stages (generation, quality assessment, trustworthiness assessment, audit, and usage) and five core components (generation methods, quality metrics, trustworthiness metrics, evaluation gap analysis, and data utilization). Quality is broken down into three pillars: validity (e.g., correctness, rule compliance), fidelity (e.g., similarity to source, diversity, style consistency), and utility (e.g., downstream task improvement, regression error). Trustworthiness is divided into fairness (bias across sensitive attributes), robustness (out‑of‑distribution and noise resilience), privacy (membership‑inference resistance, differential privacy guarantees), and safety (detection of harmful content, risk of unsafe behavior).

For each modality, the paper surveys representative LLM‑based generation methods (e.g., RedPajama‑V2, FineWeb for text; WizardMath, MetaMathQA for reasoning; TabGen‑ICL, OCTree for tables; LLM4GraphGen, GoG for graphs; Emu, Chameleon for vision‑language; ChatSUMO, AutoScenario for agents) and catalogs the evaluation metrics reported in the original works. The audit reveals a pervasive reliance on a narrow set of quality metrics (mostly validity and fidelity) and a near‑absence of trustworthiness assessments. Moreover, many studies use LLMs themselves as scoring tools, introducing model‑specific bias into the evaluation pipeline. The analysis also highlights modality imbalance: text and code enjoy richer metric suites, whereas graph, JSON, and agent data lack standardized evaluation protocols.

Key insights include: (1) the need for intrinsic evaluation frameworks that directly measure data properties rather than downstream performance; (2) the necessity to develop modality‑specific metric suites, especially for privacy, safety, and fairness; (3) the risk of bias when LLMs are used as evaluators; and (4) the importance of a “generate‑evaluate‑filter‑refine” loop to iteratively improve synthetic datasets before they enter training pipelines.

The authors propose concrete recommendations: (i) standardize quality and trustworthiness metrics and release open‑source benchmark suites; (ii) create automated auditing pipelines that combine LLM‑based checks with independent statistical tests and human validation; (iii) incorporate privacy‑preserving mechanisms (e.g., differential privacy) and safety filters into generation pipelines; (iv) encourage cross‑modality collaboration to share best practices and evaluation tools; and (v) adopt a transparent mixture design for data composition, allowing fine‑grained control over domain coverage, risk exposure, and quality signals.

In summary, the LLM Data Auditor shifts the evaluation paradigm from model‑centric to data‑centric, provides a unified taxonomy of intrinsic metrics, uncovers critical gaps in current synthetic data assessment, and offers a practical roadmap for researchers and practitioners to generate, audit, and deploy high‑quality, trustworthy synthetic data across diverse modalities.