Joint optimization of fitting & matching in multi-view reconstruction

Many standard approaches for geometric model fitting are based on pre-matched image features. Typically, such pre-matching uses only feature appearances (e.g. SIFT) and a large number of non-unique features must be discarded in order to control the false positive rate. In contrast, we solve feature matching and multi-model fitting problems in a joint optimization framework. This paper proposes several fit-&-match energy formulations based on a generalization of the assignment problem. We developed an efficient solver based on min-cost-max-flow algorithm that finds near optimal solutions. Our approach significantly increases the number of detected matches. In practice, energy-based joint fitting & matching allows to increase the distance between view-points previously restricted by robustness of local SIFT-matching and to improve the model fitting accuracy when compared to state-of-the-art multi-model fitting techniques.

💡 Research Summary

The paper “Joint optimization of fitting & matching in multi‑view reconstruction” addresses a fundamental limitation of most structure‑from‑motion pipelines: feature matching and geometric model fitting are treated as separate stages. Conventional pipelines rely on descriptor‑only matching (e.g., SIFT) and discard a large fraction of ambiguous features to keep the false‑positive rate low. This pre‑matching step becomes a bottleneck when dealing with repetitive textures, wide baseline views, or low‑texture regions, because many correct correspondences are lost before any geometric reasoning takes place.

The authors propose a unified framework that simultaneously solves feature correspondence and multi‑model fitting, which they refer to as the Fit‑&‑Match (FM) problem. The core idea is to formulate FM as an energy minimization problem that couples appearance similarity, geometric reprojection error, and a regularization term that penalizes the number of geometric models (labels) used. Two specific energy formulations are introduced:

• E₁ – a sum of unary costs (appearance + geometric error for each matched pair) and a label‑cost term β·δ_h(f) that discourages the creation of many homographies.
• E₂ – extends E₁ by adding a pairwise spatial regularizer λ·