Your Eyes Controlled the Game: Real-Time Cognitive Training Adaptation based on Eye-Tracking and Physiological Data in Virtual Reality

Cognitive training for sustained attention and working memory is vital across domains relying on robust mental capacity such as education or rehabilitation. Adaptive systems are essential, dynamically matching difficulty to user ability to maintain engagement and accelerate learning. Current adaptive systems often rely on simple performance heuristics or predict visual complexity and affect instead of cognitive load. This study presents the first implementation of real-time adaptive cognitive load control in Virtual Reality cognitive training based on eye-tracking and physiological data. We developed a bidirectional LSTM model with a self-attention mechanism, trained on eye-tracking and physiological (PPG, GSR) data from 74 participants. We deployed it in real-time with 54 participants across single-task (sustained attention) and dual-task (sustained attention + mental arithmetic) paradigms. Difficulty was adjusted dynamically based on participant self-assessment or model’s real-time cognitive load predictions. Participants showed a tendency to estimate the task as too difficult, even though they were objectively performing at their best. Over the course of a 10-minute session, both adaptation methods converged at equivalent difficulty in single-task scenarios, with no significant differences in subjective workload or game performance. However, in the dual-task conditions, the model successfully pushed users to higher difficulty levels without performance penalties or increased frustration, highlighting a user tendency to underestimate capacity under high cognitive load. Findings indicate that machine learning models may provide more objective cognitive capacity assessments than self-directed approaches, mitigating subjective performance biases and enabling more effective training by pushing users beyond subjective comfort zones toward physiologically-determined optimal challenge levels.

💡 Research Summary

This paper presents a novel real‑time adaptive system for virtual‑reality (VR) cognitive training that estimates users’ cognitive load from eye‑tracking and physiological signals (photoplethysmography, galvanic skin response) and automatically adjusts task difficulty accordingly. The authors first collected a multimodal dataset from 74 participants performing sustained‑attention tasks and dual‑task scenarios (sustained attention plus mental arithmetic) in a VR environment. Eye‑tracking metrics such as pupil diameter, fixation duration, saccade velocity, and blink patterns were extracted alongside physiological features including heart‑rate variability, mean heart rate, and skin conductance dynamics.

A bidirectional LSTM network equipped with a self‑attention layer was trained on these 25 time‑series features. The architecture captures both forward and backward temporal dependencies while allowing the model to focus on the most informative moments for load prediction. Training employed a 70/15/15 split with dropout and L2 regularization, and performance was validated using both five‑fold cross‑validation and leave‑one‑participant‑out (LOPO) testing to ensure generalisation to unseen users. The resulting model achieved 78 % accuracy on binary load classification and an R² of approximately 0.42 on continuous load regression, outperforming prior eye‑tracking‑only approaches.

For real‑time deployment, the model was integrated into a Unity‑based VR game that streams eye‑tracking and physiological data at 200 ms intervals. Difficulty is modulated along three dimensions—object speed, appearance frequency, and arithmetic problem complexity—based on either the model’s load estimate or the participant’s self‑reported difficulty rating (1‑5) collected every 30 seconds.

A second experiment involved 54 new participants who completed 10‑minute sessions of both single‑task (sustained attention) and dual‑task conditions. Two adaptation strategies were compared: (1) self‑report‑driven adaptation and (2) model‑driven adaptation. In the single‑task condition, both methods converged to a similar mean difficulty (≈3.2 on a 5‑point scale) with comparable performance (≈92 % correct) and NASA‑TLX workload scores (≈48). In the dual‑task condition, the model‑driven approach raised the mean difficulty to 4.1 while maintaining high performance (≈87 % correct) and only a modest increase in TLX (≈52), indicating that the model could safely push users beyond their self‑perceived limits without causing frustration or performance loss.

Correlation analysis revealed that participants tended to underestimate their capacity, especially under high load, suggesting a bias toward self‑efficacy conservatism. The model’s objective assessment therefore provided a more accurate gauge of available cognitive resources, enabling more effective training progression.

Limitations include the moderate size of the training dataset, potential susceptibility of PPG and GSR to environmental factors, and the fact that difficulty adjustments were limited to visual speed, object density, and arithmetic difficulty. Future work should explore richer load‑modulation parameters (e.g., working‑memory load, multimodal feedback) and robust noise‑handling for physiological signals in uncontrolled settings.

In summary, this study delivers the first end‑to‑end real‑time adaptive VR cognitive‑training system that relies solely on eye‑tracking and peripheral physiological data. By demonstrating that a deep‑learning model can objectively assess cognitive load and improve training outcomes beyond self‑report methods, the work opens a promising pathway for personalized, scalable cognitive rehabilitation and education applications.