Sensory Anticipation of Optical Flow in Mobile Robotics

In order to anticipate dangerous events, like a collision, an agent needs to make long-term predictions. However, those are challenging due to uncertainties in internal and external variables and environment dynamics. A sensorimotor model is acquired online by the mobile robot using a state-of-the-art method that learns the optical flow distribution in images, both in space and time. The learnt model is used to anticipate the optical flow up to a given time horizon and to predict an imminent collision by using reinforcement learning. We demonstrate that multi-modal predictions reduce to simpler distributions once actions are taken into account.

💡 Research Summary

This paper presents a method for enabling a mobile robot to anticipate its sensory perceptions, specifically optical flow, through an online-learned sensorimotor forward model. The core objective is to empower the robot with the ability to make long-term predictions, which is crucial for anticipating critical events like collisions amidst the uncertainties of internal states, actions, and dynamic environments.

The research is grounded in principles of developmental robotics, emphasizing autonomous learning from interaction. The authors select optical flow as the primary sensory modality because it encapsulates geometric and motion information about the scene while being largely invariant to appearance, making it a robust cue for navigation-related predictions. The robot used is a Pioneer PeopleBot equipped with a Kinect camera for dense, real-time optical flow computation and wheel encoders for proprioceptive velocity data.

A key initial analysis reveals that the conditional distribution of future optical flow given past flow (P(OF_t | OF_{t-T})) exhibits multi-modality for longer prediction horizons. The authors hypothesize and subsequently demonstrate that this complexity primarily stems from the robot’s own actions. When the action variable is accounted for, these multi-modal predictions collapse into simpler, more predictable distributions.

To model this relationship, the methodology involves learning the joint distribution of a set of input variables—past optical flow (OF_{t-T}), executed action (A_{t-T}), and past velocity (V_{t-T})—and the output variable, which is the current optical flow (OF_t). Given the high dimensionality and the need for real-time, incremental learning, the authors employ a state-of-the-art online algorithm for multivariate Gaussian Mixture Models (GMMs). This approach allows the model to learn continuously from streaming data, automatically allocating new mixture components to novel situations. Several assumptions, including a Markov property and conditional independence of variables given the mixture model, are made to ensure computational tractability for real-time prediction.

The learned GMM serves as a forward model. For prediction, given the current input state, the most activated mixture component is identified. The mean vector of the output (optical flow) dimension of that component is then used as the point prediction for the future optical flow. The paper also details a method for temporally aligning the visual and action streams to account for sensorimotor delays.

As a practical application, the framework is extended to predict imminent collisions. A temporal credit assignment mechanism is used: when a bump sensor is triggered, “collision credit” is assigned to the GMM components that were active in the frames immediately preceding the collision, with an exponential decay based on time. The future collision signal is then computed by averaging the collision values of the currently active components.

Experiments were conducted in a regular lab environment with a human operator driving the robot via joystick, using a discrete set of actions (stop, forward, backward, turn left, turn right). The performance of the GMM-based predictor was evaluated using the Average Endpoint Error (AEPE) of the predicted optical flow fields, normalized against the error of a naive predictor that assumes optical flow remains constant. Results indicated that the proposed method outperformed the naive baseline, with the relative improvement becoming more pronounced as the prediction horizon increased. This validates the model’s capability for meaningful long-term sensory anticipation.

In conclusion, this work successfully demonstrates that an incrementally learned sensorimotor model based on GMMs can enable a mobile robot to anticipate future optical flow. The significant finding is that explicitly incorporating the robot’s own actions resolves the multi-modality inherent in long-term prediction, simplifying the problem. This lays a foundation for more advanced predictive navigation and obstacle avoidance systems in robotics.