Gaze-Regularized VLMs for Ego-Centric Behavior Understanding

Eye gaze, encompassing fixations and saccades, provides critical insights into human intentions and future actions. This study introduces a gaze-regularized framework that enhances Vision Language Models (VLMs) for egocentric behavior understanding. Unlike existing methods that rely solely on visual data and overlook gaze information, our approach directly incorporates gaze information into the VLM architecture during training. By generating gaze-based queries, the model dynamically focuses on gaze-highlighted regions, while a gaze-regularization mechanism ensures the alignment of model attention with human attention patterns. To better understand how gaze can be effectively integrated into VLMs, we conducted extensive experiments exploring various strategies for incorporating gaze data. These innovations enable the prediction of future events with detailed action descriptions. Experimental results demonstrate a nearly 13 % improvement in semantic scores compared to baseline models not leveraging gaze data, highlighting the effectiveness of our approach. This work establishes a foundation for leveraging the human gaze in VLMs, significantly boosting their predictive capabilities in applications requiring accurate and robust future event prediction.

💡 Research Summary

This paper presents a novel framework that directly incorporates human eye‑gaze information into Vision‑Language Models (VLMs) to improve egocentric behavior understanding, covering both current activity comprehension and future activity prediction. The authors argue that while state‑of‑the‑art VLMs such as Flamingo, CLIP, or BLIP excel at mapping images to text, they ignore a critical source of intent information: where a person is looking. Eye‑gaze, comprising fixations and saccades, reveals the objects a person intends to interact with and the spatial relationships that guide upcoming actions.

Dataset and preprocessing
The authors build on the Ego4D dataset, which provides synchronized egocentric video and eye‑tracking data. Gaze points are rasterized into binary heatmaps, smoothed with a Gaussian filter, and down‑sampled to one frame per second to keep computation tractable. For each frame, detailed textual descriptions are generated automatically with GPT‑4V and refined through human feedback, yielding a high‑quality, fine‑grained annotation set for both observed and future frames.

Base model
The baseline VLM is an open‑source adaptation of Flamingo. A pre‑trained Vision Transformer (ViT) extracts patch embeddings from RGB frames; a Perceiver Resampler compresses these into a fixed‑size token sequence, which is then fed to the frozen language module to generate captions. This baseline receives only visual input and serves as the comparison point for all gaze‑augmented variants.

Gaze‑regularized architecture
The core contribution consists of two components:

1. Gaze‑based queries – Instead of feeding raw heatmaps as attention masks, the authors treat gaze‑augmented images (or pseudo‑gaze images generated by a separate heatmap‑prediction network) as a second visual stream. Features extracted from this stream become the query vectors (Q) in a multi‑head attention block, while the original RGB features serve as keys (K) and values (V). This design preserves the unique visual content of each frame while providing a strong signal about where attention should be directed.

2. Gaze regularizer – The attention weight distribution A produced by the Q‑K‑V operation is encouraged to match the human gaze distribution H (derived from the binary heatmaps) by minimizing the Kullback‑Leibler divergence D_KL(A‖H). This loss is added to the standard cross‑entropy loss for caption generation. The regularizer forces the model to allocate higher attention to regions that humans actually look at, without discarding information from non‑gazed areas.

When real gaze data are unavailable at test time, the auxiliary heatmap‑prediction network, trained with a cosine‑similarity loss, generates pseudo‑gaze overlays so that the same query‑generation pipeline can be used.

Training and objectives
The overall loss L = L_CE + λ·D_KL balances language modeling with gaze alignment (λ is tuned empirically). Only the vision encoder’s resampler and the cross‑attention layers are fine‑tuned; the rest of the Flamingo backbone remains frozen, allowing the authors to isolate the effect of gaze integration.

Experiments
Two tasks are evaluated:

Future Activity Prediction – Given τ_o seconds of observation (≥2 s) the model must generate textual descriptions for the next τ_a = 2 seconds.

Current Activity Understanding – Given τ_o = 1 second, the model describes what is happening now.

Metrics such as METEOR, CIDEr, and BLEU are reported, complemented by human judgments. The gaze‑regularized model achieves roughly a 13 % absolute gain in semantic scores for future prediction compared to the baseline, and a modest 2 % gain for current activity understanding. When the authors employ the fine‑grained annotations they created, an additional 12 % improvement is observed, underscoring the synergy between precise language supervision and gaze cues. Ablation studies show that removing the KL regularizer or using heatmaps directly as inputs degrades performance, confirming that both the query‑based attention and the regularization are essential.

Analysis of limitations
The approach relies on accurate gaze data; in real‑world deployment where eye‑trackers may be absent or noisy, the quality of pseudo‑gaze predictions becomes a bottleneck. Moreover, down‑sampling to one frame per second may miss rapid saccadic patterns that could be informative for very short‑term predictions. The authors suggest future work on higher‑frequency gaze capture and on extending the method to other VLM backbones (e.g., CLIP, BLIP) and multimodal signals such as audio or inertial data.

Conclusion
By embedding human visual attention directly into the attention mechanism of a Vision‑Language Model and aligning model attention with human gaze via KL divergence, the paper demonstrates a substantial boost in both understanding present egocentric actions and forecasting near‑future events. The work establishes a concrete, reproducible pipeline for leveraging gaze as a supervisory signal, opening avenues for more intent‑aware AI systems in assistive robotics, accessibility tools, and autonomous platforms.