Automatic Calibration of a Multi-Camera System with Limited Overlapping Fields of View for 3D Surgical Scene Reconstruction

The purpose of this study is to develop an automated and accurate external camera calibration method for multi-camera systems used in 3D surgical scene reconstruction (3D-SSR), eliminating the need for operator intervention or specialized expertise. The method specifically addresses the problem of limited overlapping fields of view caused by significant variations in optical zoom levels and camera locations. We contribute a novel, fast, and fully automatic calibration method based on the projection of multi-scale markers (MSMs) using a ceiling-mounted projector. MSMs consist of 2D patterns projected at varying scales, ensuring accurate extraction of well distributed point correspondences across significantly different viewpoints and zoom levels. Validation is performed using both synthetic and real data captured in a mock-up OR, with comparisons to traditional manual marker-based methods as well as markerless calibration methods. The method achieves accuracy comparable to manual, operator-dependent calibration methods while exhibiting higher robustness under conditions of significant differences in zoom levels. Additionally, we show that state-of-the-art Structure-from-Motion (SfM) pipelines are ineffective in 3D-SSR settings, even when additional texture is projected onto the OR floor. The use of a ceiling-mounted entry-level projector proves to be an effective alternative to operator-dependent, traditional marker-based methods, paving the way for fully automated 3D-SSR.

💡 Research Summary

The paper addresses a critical bottleneck in multi‑camera 3D surgical scene reconstruction (3D‑SSR): the external calibration of cameras when their fields of view overlap only minimally because of large variations in zoom level and placement. Traditional calibration methods rely on physical markers such as checkerboards, ChArUco, or AprilTag patterns that must be manually positioned within the intersecting volume of all cameras. In an operating‑room environment, this is impractical due to time constraints, the need for expert operators, and the fact that many cameras are mounted on ceiling rigs, surgical lamps, or movable arms with vastly different focal lengths. Moreover, standard Structure‑from‑Motion (SfM) pipelines fail because the scene often lacks sufficient texture and contains reflective or glossy surfaces.

To overcome these challenges, the authors propose a fully automatic calibration pipeline that uses a ceiling‑mounted entry‑level projector to project “multi‑scale markers” (MSMs) onto the operating‑room floor. An MSM consists of a single Euclidean 2‑D pattern (e.g., a square or concentric circles) that is projected repeatedly at several scale factors λ drawn from a predefined set Λ. Each projection is generated by applying a planar homography that scales the pattern about its center while preserving that center’s location. Because the center remains invariant under scaling, any camera—regardless of its distance or zoom—will see at least one projection at a size that is detectable. The projection sequence is captured as a short video (≈70 s), and a detection algorithm extracts the marker center in each frame by exploiting geometric invariants (intersection of diagonals for squares, or circle centers). These 2‑D image points constitute correspondences across cameras.

Since all detected points lie on a common planar surface (the floor), the usual essential‑matrix initialization used in SfM is degenerate. The authors therefore initialize the reconstruction by estimating a homography between two cameras that have the highest “view score” (i.e., the largest number and best distribution of correspondences). This homography is decomposed to obtain the relative rotation and translation of the initial camera pair. Subsequent cameras are added incrementally: the camera with the most correspondences to already reconstructed points is localized using a PnP algorithm, new 3‑D points (the MSM centers) are triangulated, and a bundle adjustment (BA) optimizes all camera poses and 3‑D point coordinates simultaneously. The pipeline thus mirrors standard incremental SfM but replaces the epipolar initialization with a homography‑based step tailored to coplanar points.

The method is evaluated on both synthetic and real data. In synthetic experiments, six far‑field and four near‑field cameras are arranged on two concentric circles, and three point‑distribution scenarios are tested: (1) a conventional board volume where a ChArUco board is randomly placed within a 3 m cylinder, (2) a board‑floor scenario where boards lie on the floor, and (3) a grid‑floor scenario that mimics the uniform distribution of MSM centers. Across varying noise levels, the MSM‑based calibration achieves reprojection errors comparable to or better than the gold‑standard board method, especially when scale differences are large.

In a mock‑up operating room, the authors mounted a low‑cost projector on the ceiling and used a mixture of GoPro cameras (far‑field and lamp‑mounted) and a Canon CR‑N300 (high‑zoom). After a single 70‑second projection sequence, the automatic pipeline produced an average reprojection error of 0.28 pixels (maximum 0.65 pixels). Manual ChArUco calibration failed for several cameras because the board could not be seen simultaneously, and state‑of‑the‑art SfM pipelines (Colmap, OpenMVG) did not converge due to insufficient texture and specular surfaces, even when additional texture was projected onto the floor.

Key contributions of the work include: (i) the concept of multi‑scale markers that guarantee detectable correspondences across extreme viewpoint and zoom variations, (ii) a calibration pipeline that requires no projector calibration and avoids motion blur or synchronization issues, (iii) a homography‑based initialization that handles coplanar points, and (iv) a publicly released implementation. Limitations are acknowledged: the method assumes a planar floor, so non‑planar surfaces or strong illumination changes could degrade performance; only marker centers are used, leaving potential gains from exploiting full pattern geometry. Future directions suggested are extending to non‑planar calibration surfaces, incorporating marker orientation for higher precision, and moving toward real‑time calibration suitable for dynamic surgical environments.

Overall, the study demonstrates that a simple ceiling projector combined with intelligently designed multi‑scale patterns can replace labor‑intensive manual calibration and outperform modern SfM approaches in the challenging context of 3D surgical scene reconstruction. This paves the way for more robust, automated multi‑camera systems in operating rooms, facilitating downstream applications such as intra‑operative navigation, robotic assistance, and automated workflow analysis.