A Machine Learning Approach to Recovery of Scene Geometry from Images

Recovering the 3D structure of the scene from images yields useful information for tasks such as shape and scene recognition, object detection, or motion planning and object grasping in robotics. In this thesis, we introduce a general machine learning approach called unsupervised CRF learning based on maximizing the conditional likelihood. We apply our approach to computer vision systems that recover the 3-D scene geometry from images. We focus on recovering 3D geometry from single images, stereo pairs and video sequences. Building these systems requires algorithms for doing inference as well as learning the parameters of conditional Markov random fields (MRF). Our system is trained unsupervisedly without using ground-truth labeled data. We employ a slanted-plane stereo vision model in which we use a fixed over-segmentation to segment the left image into coherent regions called superpixels, then assign a disparity plane for each superpixel. Plane parameters are estimated by solving an MRF labelling problem, through minimizing an energy fuction. We demonstrate the use of our unsupervised CRF learning algorithm for a parameterized slanted-plane stereo vision model involving shape from texture cues. Our stereo model with texture cues, only by unsupervised training, outperforms the results in related work on the same stereo dataset. In this thesis, we also formulate structure and motion estimation as an energy minimization problem, in which the model is an extension of our slanted-plane stereo vision model that also handles surface velocity. Velocity estimation is achieved by solving an MRF labeling problem using Loopy BP. Performance analysis is done using our novel evaluation metrics based on the notion of view prediction error. Experiments on road-driving stereo sequences show encouraging results.

💡 Research Summary

The thesis tackles the long‑standing problem of recovering three‑dimensional scene geometry from visual data, but it does so under a particularly challenging condition: no ground‑truth depth or motion labels are available. The core contribution is an unsupervised learning framework for conditional random fields (CRFs) that maximizes the conditional likelihood of the observed images. By iteratively alternating between estimating hidden labels (disparity planes) and updating model parameters, the method follows an EM‑like procedure that converges without any external supervision.

The geometric model employed is a slanted‑plane stereo representation. First, the left image of a stereo pair (or a single frame in a video) is over‑segmented into superpixels, each of which is assumed to lie on a single disparity plane. The plane is described by three parameters (two slopes and an intercept). Assigning a plane to every superpixel becomes a labeling problem on a Markov random field (MRF). The energy function combines a data term—measuring photometric consistency and, optionally, texture‑based shape‑from‑texture cues—with a smoothness term that penalizes abrupt changes between neighboring superpixels. Inference is performed with approximate algorithms such as Loopy Belief Propagation (LBP) or graph‑cut based methods.

A notable extension integrates surface velocity into the same framework, yielding a four‑dimensional label (disparity + 2‑D motion). This allows simultaneous structure‑from‑motion estimation from video sequences. The velocity component is also inferred via MRF labeling, again using LBP.

For evaluation, the authors introduce a novel “view prediction error” metric. After reconstructing depth and motion, the model synthesizes the next video frame from the current viewpoint; the discrepancy between the synthesized and the actual frame quantifies both geometric accuracy and temporal consistency. Experiments on road‑driving datasets (e.g., KITTI) demonstrate that the unsupervised CRF learner, even when trained only on raw image sequences, outperforms several supervised baselines. Adding texture cues improves performance in regions where color alone is ambiguous, and the motion extension accurately captures moving vehicles and pedestrians.

The work also discusses limitations: the approach heavily depends on the quality of the initial over‑segmentation, the planar assumption may break down on highly non‑planar surfaces, and LBP can be computationally demanding for real‑time applications. Future directions suggested include adaptive superpixel generation, non‑planar surface models, and more efficient inference schemes.

Overall, the thesis presents a compelling case that unsupervised CRF learning, combined with a slanted‑plane MRF formulation, can achieve state‑of‑the‑art 3D reconstruction and motion estimation without any labeled data, opening new avenues for autonomous systems operating in data‑scarce environments.