LLM Powered Social Digital Twins: A Framework for Simulating Population Behavioral Response to Policy Interventions

Predicting how populations respond to policy interventions is a fundamental challenge in computational social science and public policy. Traditional approaches rely on aggregate statistical models that capture historical correlations but lack mechanistic interpretability and struggle with novel policy scenarios. We present a general framework for constructing Social Digital Twins - virtual population replicas where Large Language Models (LLMs) serve as cognitive engines for individual agents. Each agent, characterized by demographic and psychographic attributes, receives policy signals and outputs multi-dimensional behavioral probability vectors. A calibration layer maps aggregated agent responses to observable population-level metrics, enabling validation against real-world data and deployment for counterfactual policy analysis. We instantiate this framework in the domain of pandemic response, using COVID-19 as a case study with rich observational data. On a held-out test period, our calibrated digital twin achieves a 20.7% improvement in macro-averaged prediction error over gradient boosting baselines across six behavioral categories. Counterfactual experiments demonstrate monotonic and bounded responses to policy variations, establishing behavioral plausibility. The framework is domain-agnostic: the same architecture applies to transportation policy, economic interventions, environmental regulations, or any setting where policy affects population behavior. We discuss implications for policy simulation, limitations of the approach, and directions for extending LLM-based digital twins beyond pandemic response.

💡 Research Summary

The paper introduces a novel framework called Social Digital Twins (SDTs) for predicting how populations will respond to policy interventions. An SDT consists of four components: (1) a synthetic agent population that mirrors the demographic, socioeconomic, and psychographic distribution of the target society; (2) a Large Language Model (LLM) that serves as a cognitive engine, generating multi‑dimensional behavioral probability vectors for each agent given its attributes and the current policy context; (3) a calibration layer that maps the raw LLM outputs to observable metrics (e.g., mobility percentages, consumption rates) through per‑category linear transformations with clipping; and (4) a validation protocol that enforces strict temporal train‑validation‑test splits, per‑dimension performance reporting, baseline comparisons, counterfactual sanity checks, and ablation studies.

The authors argue that traditional aggregate statistical models capture correlations but lack mechanistic interpretability, while classic agent‑based models (ABMs) require hand‑crafted decision rules that are costly to specify and domain‑specific. By contrast, LLMs have absorbed implicit models of human reasoning, preferences, and decision making from massive text corpora, allowing them to act as “neural decision makers” without explicit rule engineering. The calibration step is crucial because raw LLM probabilities are not directly comparable to real‑world percentages; learning separate slope (α) and intercept (β) parameters for each behavioral dimension aligns the simulated outputs with observed data.

To demonstrate the approach, the authors instantiate the framework for COVID‑19 pandemic response in the United Arab Emirates. The policy signal is the Oxford COVID‑19 Government Response Tracker’s Stringency Index and Government Response Index. Six behavioral dimensions correspond to Google Mobility Report categories: Retail & Recreation, Grocery & Pharmacy, Parks, Transit Stations, Workplaces, and Residential. A synthetic population of ten personas is generated, reflecting UAE’s nationality mix (10 % nationals, 90 % expatriates), occupational sectors, and risk perception levels. Gemini 2.0 Flash Lite is used as the LLM for cost efficiency; prompts embed persona attributes, date, and policy stringency, and request a JSON‑formatted probability vector.

Data are split chronologically: training (April 2020‑March 2021), validation (April‑September 2021), and testing (October 2021 onward). Baselines include a persistence model and a Gradient Boosting Machine (GBM) with lagged policy features. In the test period, the calibrated SDT achieves a macro‑averaged RMSE of 25.75 versus 32.47 for the GBM, a 20.7 % improvement. Category‑specific gains are especially pronounced for “Workplaces” (+89 % RMSE reduction) and “Retail & Recreation” (+38 %). Conversely, the GBM outperforms the SDT on low‑variance, inertia‑driven categories such as Residential and Grocery, highlighting that LLM‑driven agents excel when behavior is policy‑sensitive and semantically rich, but struggle with routine‑driven patterns lacking explicit contextual cues.

Counterfactual experiments vary the Stringency Index on a peak‑pandemic day (April 15 2020). The SDT produces monotonic increases in compliance probabilities as stringency rises, with diminishing marginal effects at the highest levels, satisfying both monotonicity and boundedness sanity checks. Ablation studies show that removing calibration inflates macro‑RMSE to 78.32, eliminating per‑category clipping modestly worsens performance, and collapsing the persona set to a single agent raises error to 32.15, confirming the importance of calibration, per‑category parameters, and population heterogeneity.

The discussion emphasizes several implications: (i) LLMs provide semantic understanding of policy‑behavior links, enabling richer simulations than purely statistical forecasts; (ii) the architecture is domain‑agnostic—by swapping policy signals, behavioral dimensions, and observable metrics, the same pipeline can model transportation mode choice under congestion pricing, energy consumption under carbon taxes, or savings behavior under interest‑rate changes; (iii) interpretability is enhanced because the LLM’s generated reasoning can be inspected, unlike black‑box time‑series models; (iv) counterfactual analysis becomes feasible, though causal identification still requires careful design.

Limitations are candidly acknowledged: (a) the experimental scale is modest (10 personas × 10 dates), raising concerns about computational cost for large‑scale deployments; (b) LLMs lack temporal memory, leading to poor modeling of inertia‑driven behaviors; (c) performance depends heavily on the quality and coverage of calibration data; (d) LLM knowledge cut‑offs may introduce anachronisms if policy contexts evolve beyond the training corpus.

Future work proposes scaling up synthetic populations, integrating autoregressive components or recurrent architectures to capture behavioral inertia, leveraging multimodal data (e.g., sensor streams, images) for richer context, and coupling the SDT with optimization frameworks for policy design.

In sum, the paper provides a compelling proof‑of‑concept that LLM‑powered social digital twins can augment traditional policy simulation tools, delivering improved predictive accuracy for policy‑sensitive behaviors while maintaining flexibility across domains. The calibration layer bridges the gap between generative LLM outputs and real‑world observables, making the approach both practical and extensible.