When Avatars Have Personality: Effects on Engagement and Communication in Immersive Medical Training

While virtual reality (VR) excels at simulating physical environments, its effectiveness for training complex interpersonal skills is limited by a lack of psychologically plausible virtual humans. This gap is particularly critical in medical education, where communication is a core clinical competency. This paper introduces a framework that integrates large language models (LLMs) into immersive VR to create medically coherent virtual patients with distinct, consistent personalities, based on a modular architecture that decouples personality from clinical data. We evaluated the system in a mixed-methods, within-subjects study with licensed physicians conducting simulated consultations. Results suggest that the approach is feasible and perceived as a rewarding and effective training enhancement. Our analysis highlights key design principles, including a “realism-verbosity paradox” and the importance of challenges being perceived as clinically authentic to support learning.

💡 Research Summary

This paper presents a novel framework that integrates large language models (LLMs) with immersive virtual reality (VR) to endow virtual patients with distinct, consistent personalities while preserving clinical fidelity. The authors argue that current VR medical training excels at replicating physical environments but falls short in reproducing the psychological and social dimensions essential for teaching complex interpersonal skills such as patient communication. To address this gap, they design a modular prompt architecture that separates four components: patient identity (behavioral rules), backstory (demographic and personal history), personality profile (communication style, affective traits), and disease card (medical facts). Each component is generated or populated using real patient inquiry data from a large‑scale Brazilian dataset, ensuring that every virtual patient presents a medically plausible case. By feeding these structured prompts sequentially to an LLM, the system produces real‑time, context‑aware dialogue that reflects both the assigned personality and the underlying disease.

Technically, the VR environment is built in Unreal Engine, featuring a photorealistic outpatient consultation room, avatar visualizations, and full head‑mounted display (HMD) support. Speech‑to‑Text (STT) transcribes the physician’s spoken input in Portuguese, which is then supplied to the LLM. The model generates a textual response conditioned on the current conversational context, the patient’s personality, and the disease card. This text is rendered into speech via a neural Text‑to‑Speech (TTS) engine and synchronized with the avatar’s lip movements, achieving a low latency of 1–2 seconds per turn. The authors also introduce a large‑scale synthetic data pipeline: thousands of simulated consultations are automatically generated, allowing quantitative assessment of personality consistency, medical accuracy, and dialogue diversity.

The empirical study follows a within‑subjects design with 20 licensed physicians. Each participant interacts with four virtual patients representing different personality archetypes (introverted vs. extroverted, cooperative vs. resistant). After each session, participants complete NASA‑TLX, System Usability Scale (SUS), and custom Likert items measuring perceived realism and variability, and they provide qualitative feedback through semi‑structured interviews. Results support the first hypothesis (feasibility of personality‑driven LLM agents) and partially confirm the second hypothesis: personality influences physicians’ questioning strategies, engagement levels, and perceived challenge. Resistant patients provoke more follow‑up questions, clarification attempts, and emotional empathy, which participants report as “challenging but valuable” for learning. Cooperative patients lead to smoother dialogue flow and quicker diagnostic focus. A recurring issue identified is the “realism‑verbosity paradox”: LLMs sometimes generate overly verbose or repetitive utterances, increasing cognitive load and reducing perceived efficiency.

The paper acknowledges several limitations. The system is currently limited to Portuguese, restricting cross‑lingual generalizability. Non‑verbal cues (facial expressions, gestures) are minimal, which may diminish social realism. The verbosity problem suggests a need for response‑length control or summarization mechanisms. Future work is outlined to incorporate multimodal affect detection, gesture synthesis, and multilingual prompt tuning, aiming to create fully embodied, socially faithful virtual patients.

In sum, the study demonstrates that LLM‑powered personality modeling can be successfully merged with high‑fidelity VR to create dynamic, personalized medical simulations. This advances the state of the art beyond scripted scenarios, offering a scalable platform for training nuanced communication skills in medicine and opening avenues for broader applications in human‑computer interaction and immersive education.