Fast Approximate Matching of Cell-Phone Videos for Robust Background Subtraction

We identify a novel instance of the background subtraction problem that focuses on extracting near-field foreground objects captured using handheld cameras. Given two user-generated videos of a scene, one with and the other without the foreground object(s), our goal is to efficiently generate an output video with only the foreground object(s) present in it. We cast this challenge as a spatio-temporal frame matching problem, and propose an efficient solution for it that exploits the temporal smoothness of the video sequences. We present theoretical analyses for the error bounds of our approach, and validate our findings using a detailed set of simulation experiments. Finally, we present the results of our approach tested on multiple real videos captured using handheld cameras, and compare them to several alternate foreground extraction approaches.

💡 Research Summary

The paper tackles a practical variant of background subtraction that arises when two handheld videos of the same scene are captured: one containing a foreground object (or set of objects) and another captured without that object. The goal is to produce a video that shows only the foreground object(s) while discarding the background. Traditional background subtraction assumes a static camera or identical camera trajectories, which does not hold for handheld recordings where the camera path is noisy and only partially overlapping.

To formalize the problem, the authors denote the foreground frame sequence as (\bar f = {f_1,\dots,f_n}) and the background sequence as (\bar b = {b_1,\dots,b_m}). A distance function (d(f_i,b_j)) quantifies visual dissimilarity between any pair of frames; in the implementation it is defined as the reciprocal of the number of SURF key‑point matches found by RANSAC. The objective is to find a mapping (\pi: