LLM-Based Social Simulations Require a Boundary

This position paper argues that LLM-based social simulations require clear boundaries to make meaningful contributions to social science. While Large Language Models (LLMs) offer promising capabilities for simulating human behavior, their tendency to produce homogeneous outputs, acting as an “average persona”, fundamentally limits their ability to capture the behavioral diversity essential for complex social dynamics. We examine why heterogeneity matters for social simulations and how current LLMs fall short, analyzing the relationship between mean alignment and variance in LLM-generated behaviors. Through a systematic review of representative studies, we find that validation practices often fail to match the heterogeneity requirements of research questions: while most papers include ground truth comparisons, fewer than half explicitly assess behavioral variance, and most that do report lower variance than human populations. We propose that researchers should: (1) match validation depth to the heterogeneity demands of their research questions, (2) explicitly report variance alongside mean alignment, and (3) constrain claims to collective-level qualitative patterns when variance is insufficient. Rather than dismissing LLM-based simulation, we advocate for a boundary-aware approach that ensures these methods contribute genuine insights to social science.

💡 Research Summary

**
This position paper argues that large‑language‑model (LLM) agents can only make meaningful contributions to social science if researchers acknowledge and respect clear methodological boundaries. The authors identify a fundamental limitation of current LLMs: they tend to generate “average‑persona” outputs, resulting in markedly low behavioral heterogeneity compared with real human populations. While many studies report high mean alignment with ground‑truth data, they rarely assess variance, and when they do, the variance of LLM‑generated behavior is consistently smaller than that observed in human samples.

The paper begins by reframing the purpose of social simulation. Rather than aiming for exact replication of a specific society or for precise prediction of future events, the authors contend that the core value of simulation lies in uncovering social patterns, generating hypotheses, and providing interpretable mechanisms that link micro‑level actions to macro‑level outcomes. In this view, the crucial requirement is not perfect individual‑level realism but sufficient heterogeneity among agents so that emergent dynamics can resemble those of real societies.

A two‑level alignment framework is introduced: (1) individual‑level alignment (does each agent behave in a human‑like way?) and (2) collective‑level alignment (do the interactions among agents reproduce realistic social phenomena?). The authors argue that collective alignment often depends more on the diversity of responses than on perfect individual realism. When agents are homogeneous, the system tends toward analytically tractable equilibria (e.g., consensus in voter models). Conversely, heterogeneous response rules generate non‑linear, unpredictable dynamics that better capture real‑world complexity.

The “average‑persona” problem is traced to the probabilistic nature of LLM decoding. Standard decoding (low temperature, greedy or top‑k sampling) concentrates probability mass on the most likely tokens, suppressing low‑probability alternatives that would introduce behavioral variety. Moreover, pre‑training data may under‑represent minority or culturally diverse behaviors, and fine‑tuning for specific tasks can further collapse the output distribution.

To substantiate these claims, the authors conduct a systematic review of 21 recent LLM‑based social simulation papers across domains such as economics, education, game theory, and network dynamics. Their analysis reveals three key patterns: (i) the majority of papers evaluate only mean alignment; (ii) fewer than half explicitly measure behavioral variance; and (iii) among those that do, the reported variance is consistently lower than that of comparable human datasets. This gap is especially pronounced in studies that require high heterogeneity (e.g., policy impact simulations, cultural diffusion models).

Based on these findings, the paper proposes three concrete recommendations:

1. Match validation depth to heterogeneity demands – Researchers should articulate how much behavioral diversity their research question requires and design validation protocols (e.g., variance, entropy, distribution of action types) accordingly.

2. Report variance alongside mean alignment – Publishing standard deviation, inter‑quartile ranges, or other dispersion metrics makes the fidelity of simulations transparent and facilitates reproducibility.

3. Constrain claim scope when variance is insufficient – If the simulated population lacks adequate heterogeneity, conclusions should be limited to qualitative, collective‑level patterns (e.g., “the model reproduces a segregation trend”) rather than attributing causal mechanisms to individual‑level behavior.

The authors also outline technical avenues to increase heterogeneity: higher temperature or nucleus sampling, ensemble of multiple LLMs, data augmentation that deliberately injects diverse cultural contexts during pre‑training, and loss functions that penalize overly concentrated output distributions during fine‑tuning.

In conclusion, the paper advocates a “boundary‑aware” stance: acknowledging LLMs’ intrinsic tendency toward homogeneity, aligning validation practices with the heterogeneity requirements of the research question, and tempering interpretive claims accordingly. By doing so, the AI community can produce simulations that genuinely illuminate social processes rather than merely echoing the average of their training data, thereby fostering a more rigorous and productive dialogue between computational modeling and social science.