Development of Immersive Virtual and Augmented Reality-Based Joint Attention Training Platform for Children with Autism

Joint Attention (JA), a crucial social skill for developing shared focus, is often impaired in children with Autism Spectrum Disorder (ASD), affecting social communication and highlighting the need for early intervention. Addressing gaps in prior research, such as limited use of immersive technology and reliance on distracting peripherals, we developed a novel JA training platform using Augmented Reality (AR) and Virtual Reality (VR) devices. The platform integrates eye gaze-based interactions to ensure participants undivided attention. To validate the platform, we conducted experiments on ASD (N=19) and Neurotypical (NT) (N=13) participants under a trained pediatric neurologist’s supervision. For quantitative analysis, we employed key measures such as the number of correct responses, the duration of establishing eye contact (s), and the duration of registering a response (s), along with correlations to CARS scores and age. Results from AR-based experiments showed NT participants registered responses significantly faster (<0.00001) than ASD participants. A correlation (Spearman coefficient=0.57, p=0.03) was found between ASD participants response time and CARS scores. A similar trend was observed in VR-based experiments. When comparing response accuracy in ASD participants across platforms, AR yielded a higher correctness rate (92.30%) than VR (69.49%), indicating AR’s greater effectiveness. These findings suggest that immersive technology can aid JA training in ASD. Future studies should explore long-term benefits and real-world applicability.

💡 Research Summary

The paper presents the design, implementation, and preliminary evaluation of an immersive Joint Attention (JA) training platform for children with Autism Spectrum Disorder (ASD) that leverages both Augmented Reality (AR) and Virtual Reality (VR) head‑mounted displays equipped with integrated eye‑tracking. Recognizing that prior JA interventions have largely relied on screen‑based or external sensor setups and have seldom compared AR and VR directly, the authors built a system in which participants interact solely through gaze. The hardware consists of a FO VE 0 VR headset and a Microsoft HoloLens 2 AR headset, each providing high‑resolution displays, six‑degree‑of‑freedom tracking, and infrared eye‑trackers. The software, developed in Unity 3D with C#, uses a Ready Player Me avatar animated via Blender and Mixamo; the avatar delivers Posner‑style cueing cues (finger pointing, head turning) toward one of two 3D objects placed on either side of it. Participants must fixate on the avatar for at least two seconds to register eye contact, then shift gaze to the target object; the system records eye‑contact duration, response registration time, and correctness in real time, storing data as JSON via a Node.js server.

A feasibility study involved 19 children diagnosed with ASD and 13 age‑matched neurotypical (NT) controls, all screened by a pediatric neurologist. Each participant completed identical tasks in both VR and AR conditions, with no prior exposure to immersive devices. Quantitative results showed that NT participants responded significantly faster than ASD participants in the AR condition (p < 0.00001). Within the ASD group, response time correlated positively with CARS scores (Spearman ρ = 0.57, p = 0.03), indicating that higher autism severity is associated with slower gaze‑based responses. Accuracy was markedly higher in AR (92.30 % correct) than in VR (69.49 % correct), suggesting that AR’s spatial alignment with the real world reduces cognitive load and improves task performance. Similar trends were observed in the VR experiments, though the magnitude of differences was smaller.

The study’s contributions are threefold: (1) a novel gaze‑only interaction paradigm that eliminates peripheral distractions; (2) a side‑by‑side comparison of AR and VR platforms for JA training, providing empirical evidence that AR may be more effective for this population; and (3) an integrated data‑logging framework that captures fine‑grained behavioral metrics for future analysis.

Limitations include the modest sample size, narrow age range, and the fact that only a single training session was administered, precluding conclusions about long‑term learning or transfer to real‑world social interactions. Hardware differences in eye‑tracking precision between the two headsets could also confound platform comparisons. The authors recommend future work involving longitudinal multi‑session trials, larger and more diverse cohorts, and multimodal measurements (e.g., EEG, heart‑rate variability) to better understand the neurophysiological underpinnings of gaze‑based JA training.

In summary, this research demonstrates that immersive AR/VR technologies, when coupled with eye‑tracking‑driven interaction, can feasibly and effectively support joint‑attention skill development in children with ASD, laying groundwork for more extensive clinical investigations and eventual real‑world deployment.