Alignment among Language, Vision and Action Representations

A fundamental question in cognitive science and AI concerns whether different learning modalities: language, vision, and action, give rise to distinct or shared internal representations. Traditional views assume that models trained on different data types develop specialized, non-transferable representations. However, recent evidence suggests unexpected convergence: models optimized for distinct tasks may develop similar representational geometries. We investigate whether this convergence extends to embodied action learning by training a transformer-based agent to execute goal-directed behaviors in response to natural language instructions. Using behavioral cloning on the BabyAI platform, we generated action-grounded language embeddings shaped exclusively by sensorimotor control requirements. We then compared these representations with those extracted from state-of-the-art large language models (LLaMA, Qwen, DeepSeek, BERT) and vision-language models (CLIP, BLIP). Despite substantial differences in training data, modality, and objectives, we observed robust cross-modal alignment. Action representations aligned strongly with decoder-only language models and BLIP (precision@15: 0.70-0.73), approaching the alignment observed among language models themselves. Alignment with CLIP and BERT was significantly weaker. These findings indicate that linguistic, visual, and action representations converge toward partially shared semantic structures, supporting modality-independent semantic organization and highlighting potential for cross-domain transfer in embodied AI systems.

💡 Research Summary

This paper investigates whether internal representations learned from language, vision, and embodied action converge toward a shared semantic space. The authors train a transformer‑based agent on the BabyAI platform using behavioral cloning. The agent receives a natural‑language instruction and a local RGB view of a 2D grid world, and must output one of six primitive actions. Crucially, the language token embeddings are initialized randomly and are shaped solely by the action‑prediction loss, producing “action‑grounded language embeddings.”

To assess cross‑modal alignment, the study compares these embeddings with those extracted from state‑of‑the‑art large language models (LLaMA‑7B, Qwen‑7B, DeepSeek‑7B, BERT‑base) and vision‑language models (CLIP‑ViT‑B/32, BLIP‑ViT‑L). All embeddings are projected to a common 128‑dimensional space, and similarity is measured using cosine‑based precision@15. Results show that action embeddings align strongly with decoder‑only LLMs and BLIP, achieving precision@15 scores of 0.70–0.73, comparable to the alignment observed among the language models themselves. Alignment with CLIP and BERT is substantially weaker (≈0.45 or lower).

These findings challenge the traditional view that representations are modality‑specific and non‑transferable. The strong alignment with BLIP suggests that models trained on combined image‑text data capture semantic structures that are also useful for sensorimotor control. The authors argue that the shared structures likely encode concepts such as object identity, color, and spatial relations, which are necessary both for language understanding and for goal‑directed action.

The paper also discusses limitations: experiments are confined to a simple 2D environment with a limited action set, and the generalization to richer 3D or continuous control domains remains open. Future work is proposed to explore more complex robotic simulations, multi‑agent scenarios, and bidirectional mapping techniques that could enable seamless transfer between language, vision, and action modalities. Overall, the study provides compelling evidence that linguistic, visual, and action representations can converge onto partially shared semantic spaces, opening avenues for cross‑domain transfer in embodied AI systems.