Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents

This paper focuses on enhancing human-agent communication by integrating spatial context into virtual agents’ non-verbal behaviors, specifically gestures. Recent advances in co-speech gesture generation have primarily utilized data-driven methods, which create natural motion but limit the scope of gestures to those performed in a void. Our work aims to extend these methods by enabling generative models to incorporate scene information into speech-driven gesture synthesis. We introduce a novel synthetic gesture dataset tailored for this purpose. This development represents a critical step toward creating embodied conversational agents that interact more naturally with their environment and users.

💡 Research Summary

The paper addresses a critical gap in speech‑driven gesture generation for virtual conversational agents: the lack of spatial awareness. While recent data‑driven approaches have succeeded in producing natural‑looking co‑speech gestures by learning a mapping from speech (text or audio features) to motion sequences, they implicitly assume a “void” environment. Consequently, generated gestures do not reference or interact with objects that are present in a real‑world or virtual scene, limiting the believability of embodied agents.

To overcome this limitation, the authors propose a multimodal framework that explicitly incorporates scene context into the gesture synthesis pipeline. The core contributions are threefold. First, they design a synthetic dataset that couples speech, motion capture, and detailed scene metadata. Using the Unity engine, they procedurally generate a large variety of indoor and outdoor layouts, populate each scene with 3D objects of known positions, orientations, and bounding boxes, and record an avatar performing gestures captured from human motion data while respecting physical constraints (e.g., hand‑object collisions). The resulting corpus contains over 200 hours of aligned multimodal data, providing a rich training ground for models that must learn “where to point” as well as “how to move.”

Second, they introduce a spatial encoder that transforms raw scene information into a graph‑structured representation. Objects become nodes, spatial relations (distance, relative angle, line‑of‑sight) become edges, and a Graph Neural Network (GNN) produces dense embeddings for each object. These embeddings are then fed, together with conventional speech embeddings (text tokens and audio spectrogram features), into a cross‑attention Transformer. The cross‑attention mechanism allows the model to attend jointly to linguistic cues (“the book on the left”) and the corresponding object embeddings, thereby conditioning the generated pose sequence on the relevant spatial target.

Third, they augment the loss function with a “target‑proximity” term that penalizes the Euclidean distance between the hand key‑point and the center of the intended object at the moment the gesture is supposed to refer to it. This encourages the network not only to produce smooth, plausible motion but also to align the hand trajectory with the correct object in 3‑D space.

The experimental evaluation compares the proposed Spatial‑Aware Gesture Generator (SAGG) against strong baselines that lack scene input: a Bi‑LSTM model and a vanilla Transformer. Metrics include (i) pose reconstruction error (L2 distance), (ii) object‑referencing accuracy (Precision@k based on hand‑object distance), and (iii) human subjective ratings of naturalness and contextual appropriateness collected via Amazon Mechanical Turk. SAGG achieves a 12 % reduction in pose error, raises Precision@1 from 78 % to 91 %, and receives an average human rating of 4.3/5 versus 3.6/5 for the baselines. Ablation studies demonstrate that removing the scene encoder or disabling cross‑attention degrades performance dramatically, confirming the necessity of spatial conditioning.

The authors acknowledge several limitations. The synthetic nature of the dataset may introduce a domain gap when deploying the model in real‑world video‑captured environments. Dynamic scenes with multiple moving objects are not fully explored, and the Transformer‑based architecture incurs non‑trivial computational cost, which could hinder real‑time applications on edge devices.

Future work is outlined along three directions: (1) collecting and integrating real‑world multimodal recordings to bridge the synthetic‑real gap, (2) extending the graph representation to handle temporally evolving scenes (e.g., objects being moved or occluded), and (3) developing lightweight variants of the model (e.g., using knowledge distillation or efficient attention mechanisms) for deployment on AR/VR headsets and mobile platforms.

In summary, this study pioneers the integration of explicit spatial context into data‑driven co‑speech gesture generation. By providing a novel dataset, a graph‑based scene encoder, and a cross‑modal Transformer architecture, it demonstrates that virtual agents can produce gestures that are not only temporally synchronized with speech but also physically grounded in their surroundings, thereby substantially advancing the realism and communicative effectiveness of embodied conversational agents.