Robust Multi-view Camera Calibration from Dense Matches

Estimating camera intrinsics and extrinsics is a fundamental problem in computer vision, and while advances in structure-from-motion (SfM) have improved accuracy and robustness, open challenges remain. In this paper, we introduce a robust method for pose estimation and calibration. We consider a set of rigid cameras, each observing the scene from a different perspective, which is a typical camera setup in animal behavior studies and forensic analysis of surveillance footage. Specifically, we analyse the individual components in a structure-from-motion (SfM) pipeline, and identify design choices that improve accuracy. Our main contributions are: (1) we investigate how to best subsample the predicted correspondences from a dense matcher to leverage them in the estimation process. (2) We investigate selection criteria for how to add the views incrementally. In a rigorous quantitative evaluation, we show the effectiveness of our changes, especially for cameras with strong radial distortion (79.9% ours vs. 40.4 vanilla VGGT). Finally, we demonstrate our correspondence subsampling in a global SfM setting where we initialize the poses using VGGT. The proposed pipeline generalizes across a wide range of camera setups, and could thus become a useful tool for animal behavior and forensic analysis.

💡 Research Summary

The paper tackles the problem of self‑calibration for a rigid multi‑camera rig when no calibration pattern is available, a scenario common in animal‑behavior studies (camera traps) and forensic video analysis. While recent structure‑from‑motion (SfM) pipelines such as COLMAP, VGGT, and VGG‑SfM have achieved impressive results on standard datasets, they still struggle with cameras that exhibit strong radial distortion (e.g., fisheye lenses) and with limited baseline configurations. The authors therefore revisit two crucial stages of an SfM pipeline—correspondence selection and view‑order determination—and propose a set of design choices that dramatically improve robustness and accuracy.

Dense correspondence sampling
The method starts from a dense matcher (RoMa) that provides a per‑pixel warp map and a confidence map. To filter out unreliable matches, the authors introduce an n‑cycle test: a pixel is kept only if applying the warp n times returns it to its original location. Two‑cycle (forward‑backward) matches are always retained; for higher‑order cycles (n > 2) the cycle must close and all its (n‑1) sub‑cycles must also close. This strict criterion yields a set of highly reliable dense correspondences.

Because using all dense matches would bias the reconstruction toward regions of high texture and would be computationally prohibitive, the authors perform a two‑stage grid‑based subsampling. In the first stage a fine 5 × 5 pixel grid is overlaid on the warp output; each cell contributes at most one randomly chosen correspondence that satisfies the n‑cycle condition. This reduces the number of matches dramatically while preserving spatial coverage. In the second stage a coarser grid is applied, and within each cell a single correspondence is selected based on a score that reflects the quality of the underlying triangulation geometry.

Score function
The score is derived from the triangulation angle θ of a pair of observations. Inspired by NASA Ames Stereo Pipeline heuristics, the authors model the ideal angle distribution as a symmetric Gaussian centered at α = 30° with σ = 20°, truncated by a boundary term B(θ) that forces the score to zero as θ approaches 0° or 180°. The final per‑pair score is

f(θ) = p · B(θ) · G(θ)

where p normalizes the maximum to 1. For an n‑cycle, the scores of all constituent pairs are computed and the top‑k scores (k ∈ {0,1,2,3}) are summed to obtain the correspondence’s overall score. This encourages matches that belong to long, well‑conditioned tracks while still allowing uniform sampling (k = 0) for comparison.

Initial focal length estimation
Before any bundle adjustment, the pipeline needs a reasonable estimate of each camera’s focal length. The authors adopt the approach of Mendonca and Cipolla (1999), which exploits the property that the two largest singular values of the essential matrix should be equal and the smallest should be zero. They formulate a residual based on the singular values of E = Kᵀ F K and solve a non‑linear least‑squares problem over the focal lengths (assuming fx = fy). Weights wᵢⱼ are proportional to the fraction of matches used to compute each fundamental matrix, and the fundamental matrices themselves are obtained via OpenCV’s MAGSA implementation.

View‑order selection
The order in which views are added in incremental SfM strongly influences convergence. The authors compute the cycle scores (Section 2) for all three‑view cycles. The initial three cameras are chosen as the highest‑scoring cycle. Thereafter, each candidate camera is evaluated by the increase it would cause in the total score of all three‑cycles that involve the already‑selected set plus the candidate. The camera yielding the largest increase is added. For the very first pair of cameras, a brute‑force evaluation of the three possible pairs (1‑2, 1‑3, 2‑3) is performed, using a cost that combines the number of points in front of both cameras and the sum of reprojection errors. This mitigates the risk of poor local minima that can arise in highly concave or convex scenes.

Global SfM integration
After the incremental stage, the authors demonstrate that their subsampling scheme can also be used in a global SfM setting. They initialize camera poses with VGGT, then replace the dense matches with the subsampled set and run a single global bundle adjustment. This hybrid approach leverages the speed of deep‑learning pose predictors while correcting their distortion‑related errors with classic geometric optimization.

Experimental validation
The method is evaluated on two application domains:

1. Animal‑behavior dataset (PFERD) – a collection of multi‑view recordings of horses. The proposed pipeline produces dense point clouds with accurate skeletal reconstructions, outperforming vanilla VGGT in both reprojection error and 3‑D shape fidelity.

2. Forensic scenario – simulated crime scenes captured by four mobile phones with strong radial distortion. The authors report a calibration success rate of 79.9 % compared to 40.4 % for vanilla VGGT, and show qualitatively better localization of objects and people.

Quantitative metrics include mean reprojection error, average triangulation angle, and intrinsic parameter deviation. Across all experiments, the proposed pipeline reduces reprojection error by roughly 30 % and yields more stable focal length estimates, especially for lenses with distortion coefficients exceeding typical values.

Complexity and implementation
The hierarchical sampling algorithm runs in O(|S₁|·max_iter·(N‑1)) time, where |S₁| is the number of matches after the fine‑grid stage. The authors limit N to 4 and max_iter to three refinements, keeping runtime linear in the number of images. All components are implemented in Python/C++ and the code, along with pretrained models for RoMa, is released publicly.

Limitations and future work
The authors acknowledge that scenes lacking texture (e.g., white walls) may produce insufficient dense matches, and that scaling to rigs with dozens of cameras could require additional hierarchical clustering. They suggest integrating learned confidence measures and adaptive grid sizes as possible extensions.

Conclusion
By carefully filtering dense correspondences with n‑cycle closure, applying a geometry‑aware scoring function, and selecting view order based on cycle scores, the paper delivers a robust multi‑view calibration pipeline that works well even with heavily distorted lenses. The approach bridges the gap between modern deep‑learning SfM methods and classical geometric optimization, offering a practical tool for fields such as wildlife monitoring and forensic video reconstruction.