Virtual Reflections on a Dynamic 2D Eye Model Improve Spatial Reference Identification

The visible orientation of human eyes creates some transparency about people’s spatial attention and other mental states. This leads to a dual role of the eyes as a means of sensing and communication. Accordingly, artificial eye models are being explored as communication media in human-machine interaction scenarios. One challenge in the use of eye models for communication consists of resolving spatial reference ambiguities, especially for screen-based models. To address this challenge, we introduce an approach that incorporates reflection-like features that are contingent on the movements of artificial eyes. We conducted a user study with 30 participants in which participants had to use spatial references provided by dynamic eye models to advance in a fast-paced group interaction task. Compared to a non-reflective eye model and a pure reflection mode, the superimposition of screen-based eyes with gaze-contingent virtual reflections resulted in a higher identification accuracy and user experience, suggesting a synergistic benefit.

💡 Research Summary

The paper addresses the problem of spatial reference ambiguity when using screen‑based artificial eye models for human‑machine communication. Human eyes naturally convey both gaze direction and a subtle reflection of the surrounding scene, but 2D displays lack depth cues, making it difficult for observers to infer where the eye is looking, especially in group interactions. To overcome this limitation, the authors introduce “Mirror Eyes,” a concept that superimposes a gaze‑contingent virtual mirror image onto a dynamic 2D eye model. The mirror image is a horizontally flipped view of the camera feed, placed near the pupil so that the observer can see both the eye’s orientation and a miniature representation of the attended area in a single glance.

The prototype consists of four interconnected modules implemented in Python with OpenCV and MediaPipe: (1) a dynamic eye model that moves pupil and iris in real time and estimates object distance using camera parameters; (2) an attention module that tracks multiple faces and continuously updates target coordinates; (3) a virtual‑reflection module that creates the mirrored overlay, applying spatial shifts defined by equations (1)–(4) to keep the reflection aligned with the pupil while preserving image resolution; and (4) integration software that runs the pipeline at interactive frame rates. Depth cues are simulated by varying the interpupillary distance (IPD) based on the estimated focus distance, thereby mimicking the natural eye‑vergence angle (EVA).

To evaluate the approach, a user study with 30 participants was conducted. Participants played a fast‑paced, modified version of the Japanese word‑chain game “Shiritori” in groups of three. In each trial, the artificial eye (under one of three display conditions) looked at one participant, who then had to press a button within three seconds to indicate that they had been selected. The three conditions were: (a) Eyes‑Only (dynamic eye without any reflection), (b) Mirrors‑On (dynamic eye with the virtual mirror overlay), and (c) Pure‑Reflection (mirror overlay without eye features). Accuracy (correct button press) and reaction time (time to press) were recorded, and post‑experiment questionnaires measured user experience, perceived workload, and immersion.

Statistical analysis (repeated‑measures ANOVA) showed that the Mirrors‑On condition yielded a statistically significant increase in identification accuracy (≈18 percentage points higher than Eyes‑Only) and a reduction in reaction time (≈0.42 seconds faster). The Pure‑Reflection condition performed similarly to Eyes‑Only in accuracy but received lower usability scores, suggesting that the presence of eye features (pupil, iris) is important for intuitive interpretation. Consequently, the first hypothesis (accuracy improvement) and the third hypothesis (enhanced user experience) were supported, while the second hypothesis (speed reduction) was contradicted—the Mirror Eyes actually sped up decision making.

The authors discuss several implications. By embedding a miniature view of the attended region directly within the eye, the system supplies a depth cue that compensates for the 2D display’s lack of stereopsis, enabling observers to infer both direction and approximate distance of the target. The approach also reduces the cognitive load of a two‑step inference (gaze → target) to a single step. Limitations include the current design’s reliance on a single observer viewpoint; in multi‑user scenarios, conflicting perspectives could cause visual ambiguity. Moreover, the distance estimation depends on accurate camera calibration and known object sizes, which may be problematic in uncontrolled environments. The visual complexity introduced by the overlay could increase perceptual load, a factor that warrants further physiological measurement (e.g., eye‑tracking, EEG).

Future work is outlined to integrate Mirror Eyes with AR/VR headsets, providing true binocular depth cues, and to employ machine‑learning‑based face recognition for smoother attention shifts. Extending the concept to convey affective states or to support collaborative tasks with multiple simultaneous observers are also proposed.

In summary, the study demonstrates that virtual reflections superimposed on dynamic 2D eye models can significantly improve spatial reference identification and user experience in fast‑paced group interactions, offering a promising direction for more effective visual communication in human‑machine interfaces.