Autonomously Learning to Visually Detect Where Manipulation Will Succeed

Visual features can help predict if a manipulation behavior will succeed at a given location. For example, the success of a behavior that flips light switches depends on the location of the switch. Within this paper, we present methods that enable a mobile manipulator to autonomously learn a function that takes an RGB image and a registered 3D point cloud as input and returns a 3D location at which a manipulation behavior is likely to succeed. Given a pair of manipulation behaviors that can change the state of the world between two sets (e.g., light switch up and light switch down), classifiers that detect when each behavior has been successful, and an initial hint as to where one of the behaviors will be successful, the robot autonomously trains a pair of support vector machine (SVM) classifiers by trying out the behaviors at locations in the world and observing the results. When an image feature vector associated with a 3D location is provided as input to one of the SVMs, the SVM predicts if the associated manipulation behavior will be successful at the 3D location. To evaluate our approach, we performed experiments with a PR2 robot from Willow Garage in a simulated home using behaviors that flip a light switch, push a rocker-type light switch, and operate a drawer. By using active learning, the robot efficiently learned SVMs that enabled it to consistently succeed at these tasks. After training, the robot also continued to learn in order to adapt in the event of failure.

💡 Research Summary

The paper introduces a framework that enables a mobile manipulator to autonomously learn a visual function mapping RGB‑D observations to 3‑D locations where a given manipulation behavior is likely to succeed. The approach hinges on three ingredients: (1) a pair of complementary behaviors that can change the state of the same object (e.g., “flip switch up” and “flip switch down”), (2) binary success classifiers that can automatically label an execution as successful or failed based on sensor feedback, and (3) an initial hint—a rough estimate of where one of the behaviors might work. Starting from the hint, the robot repeatedly selects a 3‑D point, executes one of the behaviors, and uses the success classifier to obtain a label. The RGB image and registered point cloud at that point are transformed into a feature vector (color histograms, texture descriptors, depth‑based shape cues) which, together with the label, is added to the training set of a support vector machine (SVM).

Active learning drives the selection of query points: the robot queries the location where the current SVM’s decision margin is smallest, i.e., where the model is most uncertain. Because the two behaviors are complementary, a failure of one often yields a successful trial of the other, providing additional labeled data and reducing bias. Over successive iterations the SVM converges to a reliable predictor of manipulation success.

The method was evaluated on a PR2 robot in a simulated home environment. Three tasks were tested: (a) flipping a toggle light switch, (b) pushing a rocker‑type switch, and (c) pulling a drawer open. For each task a pair of opposite actions and corresponding success classifiers were implemented. With only about 15–20 exploratory trials per task, the robot achieved success rates above 80 % and continued to refine its model when the environment changed (e.g., the switch was moved).

Key contributions include (i) a self‑supervised learning loop that generates its own labeled data, (ii) an uncertainty‑driven active learning strategy that minimizes the number of required trials, and (iii) a demonstration that the learned visual predictor can be used repeatedly for reliable manipulation in a realistic domestic setting.

Limitations are acknowledged: the linear SVM may not capture highly non‑linear visual‑action relationships, the approach relies on a reasonably accurate initial hint, and the behaviors themselves are fixed primitives rather than being optimized during learning. Future work is suggested to incorporate deep feature extractors, non‑linear kernels or neural networks, to jointly learn action parameters (force, trajectory) together with the visual predictor, and to extend the framework to long‑term adaptation and multi‑robot collaboration.