Bridging Speech, Emotion, and Motion: a VLM-based Multimodal Edge-deployable Framework for Humanoid Robots

Effective human-robot interaction requires emotionally rich multimodal expressions, yet most humanoid robots lack coordinated speech, facial expressions, and gestures. Meanwhile, real-world deployment demands on-device solutions that can operate autonomously without continuous cloud connectivity. To bridging \underline{\textit{S}}peech, \underline{\textit{E}}motion, and \underline{\textit{M}}otion, we present \textit{SeM$^2$}, a Vision Language Model-based framework that orchestrates emotionally coherent multimodal interactions through three key components: a multimodal perception module capturing user contextual cues, a Chain-of-Thought reasoning for response planning, and a novel Semantic-Sequence Aligning Mechanism (SSAM) that ensures precise temporal coordination between verbal content and physical expressions. We implement both cloud-based and \underline{\textit{e}}dge-deployed versions (\textit{SeM$^2_e$}), with the latter knowledge distilled to operate efficiently on edge hardware while maintaining 95% of the relative performance. Comprehensive evaluations demonstrate that our approach significantly outperforms unimodal baselines in naturalness, emotional clarity, and modal coherence, advancing socially expressive humanoid robotics for diverse real-world environments.

💡 Research Summary

The paper introduces SeM², a Vision‑Language Model (VLM)‑driven framework that enables humanoid robots to produce emotionally coherent, multimodal interactions encompassing speech, facial expressions, and gestures. The authors identify a gap in current human‑robot interaction (HRI) research: while robots have advanced in locomotion and manipulation, coordinated emotional expression across modalities remains under‑explored, especially for on‑device deployment without continuous cloud connectivity.

Core Architecture
SeM² consists of three tightly coupled components:

1. Multimodal Perception Module – It fuses audio and visual streams. Speech is processed by SenseVoice, which provides both transcription and affective cues. Facial landmarks are extracted using YOLOv8‑face. The combined perception results are formatted into a structured prompt that feeds a large VLM (the authors use a GPT‑4‑style model).

2. Chain‑of‑Thought (CoT) Reasoning – Rather than a single pass generation, the VLM is guided through a step‑by‑step reasoning chain that explicitly considers semantic consistency among language, facial expression, and motion. The prompt encodes persona constraints, embodiment limits (available expression and motion primitives), and emotional alignment requirements.

3. Semantic‑Sequence Aligning Mechanism (SSAM) – The novelty lies in temporally aligning textual tokens with expression/motion primitives. Each word wᵢ is assigned a predicted speech duration τ(wᵢ) scaled by a speed factor α. Semantic relevance S(wᵢ, aⱼ) between the word and a candidate action/expression aⱼ is computed via cosine similarity of pretrained embeddings (BERT‑base). Pairs exceeding a threshold θ are kept. A dynamic‑programming optimization then selects execution times T(aⱼ) that satisfy both semantic relevance and temporal constraints (no overlap, minimal latency). This yields a schedule where, for example, a “surprised” utterance triggers widened eyes and a hand raise precisely at the word “wow”.

Edge Deployment (SeM²ₑ)
To meet real‑time, on‑device requirements, the authors distill knowledge from a cloud‑based teacher model (GPT‑4‑o) into a lightweight student model (MiniCPM‑8B). They curate 11,500 multimodal samples, apply deduplication, and perform supervised fine‑tuning (SFT) followed by 4‑bit quantization. The resulting edge model runs on CPU‑only embedded boards at ~20 fps, consuming ~2 GB RAM, and retains ~95 % of the cloud version’s performance on key metrics.

Experimental Evaluation
Evaluation combines automatic metrics (BLEU‑like text similarity, timing error for expressions/motions, emotion label accuracy) and human user studies (30 participants rating naturalness, emotional clarity, and overall satisfaction on a 5‑point Likert scale). SeM² outperforms unimodal baselines (speech‑only, expression‑only, motion‑only) by 18‑27 % across all metrics. Ablation studies show that removing SSAM dramatically increases timing errors (over 2×) and reduces perceived emotional clarity, confirming SSAM’s central role.

Limitations & Future Work
The current system relies on a predefined library of facial and motion scripts, limiting the generation of highly nuanced or free‑form gestures. SSAM’s thresholds (θ, δ) are dataset‑specific and may require retuning for new domains. The edge model’s memory footprint, while acceptable for many platforms, remains a bottleneck for ultra‑low‑power robots. Future directions include learning gesture primitives directly from VLM outputs, meta‑learning to adapt SSAM parameters on‑the‑fly, and further model compression techniques.

Conclusion
SeM² demonstrates that a large VLM can serve as a unified reasoning engine for multimodal HRI, and that a principled semantic‑temporal alignment mechanism can produce fluid, emotionally resonant robot behavior. By delivering both cloud‑based and edge‑optimized versions, the work bridges the gap between research prototypes and deployable socially expressive humanoid robots for real‑world environments.