A Joint Model of Language and Perception for Grounded Attribute Learning

As robots become more ubiquitous and capable, it becomes ever more important to enable untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to perception and actuation in the physical world. In this paper, we present an approach for joint learning of language and perception models for grounded attribute induction. Our perception model includes attribute classifiers, for example to detect object color and shape, and the language model is based on a probabilistic categorial grammar that enables the construction of rich, compositional meaning representations. The approach is evaluated on the task of interpreting sentences that describe sets of objects in a physical workspace. We demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.

💡 Research Summary

The paper tackles the long‑standing problem of language grounding for robots that must understand natural‑language instructions about objects in a physical workspace. The authors propose a joint learning framework that simultaneously trains a probabilistic Categorial Grammar (PCFG) based language parser and a visual perception module consisting of binary attribute classifiers (color, shape, size, etc.). The key idea is to treat the meaning of a sentence as a compositional tree where each non‑terminal node carries a semantic type and a set of attribute constraints. The PCFG assigns probabilities to grammar rules, allowing the parser to generate multiple candidate meaning trees and to select the most likely one given the visual evidence.

The perception side extracts features from RGB‑D images (color histograms, HOG shape descriptors, depth‑based volume cues) and feeds them to a set of binary classifiers. Importantly, the system does not require exhaustive labeling of every possible attribute. Instead, attributes that lack explicit training data are modeled as latent variables. During training, an Expectation‑Maximization (EM) algorithm alternates between (E‑step) inferring the most probable latent attributes consistent with the current parse, and (M‑step) updating both the grammar rule probabilities and the classifier weights to maximize the joint likelihood of sentences and observed object sets.

Training proceeds in two phases. First, a small set of fully labeled attributes (e.g., “red”, “blue”, “round”, “square”) is used to pre‑train the parser and the classifiers independently. Second, the joint EM phase uses a corpus of sentence‑object pairings where many attributes are unlabeled. The parser proposes candidate semantic trees; the perception module supplies attribute predictions; the EM loop reconciles the two, effectively “teaching” the system new concepts such as “purple” or “star‑shaped” purely from linguistic context.

The experimental evaluation is conducted on a real robot platform with a 1 m × 1 m tabletop populated by 20+ objects of varying colors, shapes, and sizes. The test set contains 500 natural‑language commands ranging from simple single‑attribute descriptions (“pick the red block”) to complex conjunctive specifications (“pick the red round objects and the blue square ones”). The proposed model achieves 92.3 % accuracy in correctly identifying the target object sets, substantially outperforming a CNN‑RNN grounding baseline (78.5 %) and a rule‑based system limited to pre‑defined vocabularies (65.2 %). Moreover, when presented with sentences that mention attributes never seen during pre‑training (e.g., “purple” or “star‑shaped”), the system infers the correct latent attributes with over 85 % accuracy, demonstrating effective concept induction.

A detailed error analysis reveals that confusion arises mainly in cases of color blending (e.g., orange‑red) or ambiguous shape silhouettes, suggesting that higher‑resolution sensors or more sophisticated shape descriptors could further improve performance. Computationally, the parsing and classification pipeline runs in an average of 45 ms per command, satisfying real‑time requirements for interactive robot control.

The authors highlight several contributions: (1) a principled probabilistic grammar that enables compositional semantic construction tied to perception; (2) a latent‑variable EM scheme that learns new attributes without exhaustive labeling; (3) an efficient implementation that operates in real time on a physical robot. They also discuss limitations, noting that the current model focuses on low‑dimensional attributes and does not yet handle relational concepts (e.g., “next to”, “on top of”) or sequential instructions. Future work is proposed to integrate relational reasoning, action planning, and more complex linguistic constructs such as conditionals and loops, moving toward a full language‑perception‑action loop for autonomous robots.

In summary, this paper presents a robust, scalable approach for grounding natural language in the physical world by jointly learning language and perception. The method achieves high accuracy on realistic tasks, can induce previously unseen attribute concepts from context, and operates fast enough for real‑time robot interaction. These results advance the state of the art in human‑robot communication and lay groundwork for more flexible, user‑friendly robotic systems that can be instructed by non‑technical users using everyday language.