Self-Supervised Contrastive Embedding Adaptation for Endoscopic Image Matching

Accurate spatial understanding is essential for image-guided surgery, augmented reality integration and context awareness. In minimally invasive procedures, where visual input is the sole intraoperative modality, establishing precise pixel-level correspondences between endoscopic frames is critical for 3D reconstruction, camera tracking, and scene interpretation. However, the surgical domain presents distinct challenges: weak perspective cues, non-Lambertian tissue reflections, and complex, deformable anatomy degrade the performance of conventional computer vision techniques. While Deep Learning models have shown strong performance in natural scenes, their features are not inherently suited for fine-grained matching in surgical images and require targeted adaptation to meet the demands of this domain. This research presents a novel Deep Learning pipeline for establishing feature correspondences in endoscopic image pairs, alongside a self-supervised optimization framework for model training. The proposed methodology leverages a novel-view synthesis pipeline to generate ground-truth inlier correspondences, subsequently utilized for mining triplets within a contrastive learning paradigm. Through this self-supervised approach, we augment the DINOv2 backbone with an additional Transformer layer, specifically optimized to produce embeddings that facilitate direct matching through cosine similarity thresholding. Experimental evaluation demonstrates that our pipeline surpasses state-of-the-art methodologies on the SCARED datasets improved matching precision and lower epipolar error compared to the related work. The proposed framework constitutes a valuable contribution toward enabling more accurate high-level computer vision applications in surgical endoscopy.

💡 Research Summary

This paper addresses the challenging problem of establishing accurate pixel‑level correspondences between endoscopic video frames, which is essential for 3D reconstruction, camera tracking, and augmented reality in minimally invasive surgery. Conventional feature detectors such as SIFT, SURF, and ORB perform poorly in the surgical environment due to weak textures, specular reflections, tissue deformation, and variable illumination. Recent transformer‑based matchers (e.g., SuperGlue, LoFTR) improve robustness but still require large annotated datasets and struggle with the unique visual characteristics of endoscopic imagery.

The authors propose a self‑supervised contrastive learning pipeline that adapts a pretrained DINOv2 vision transformer to the specific demands of endoscopic matching. The core architecture consists of two parts: (1) a frozen DINOv2 backbone Φ that extracts high‑level semantic feature maps from each input RGB image, and (2) an additional trainable transformer layer Ψ that refines these semantic embeddings into discriminative, locally distinctive descriptors suitable for dense matching. The output descriptors Ms and Mt (one per image patch) are used to compute a cosine similarity matrix S, from which one‑to‑one correspondences are obtained by mutual argmax and filtered with a high similarity threshold (≥ 0.95).

Because manually annotated correspondences are unavailable for endoscopic data, the authors introduce a novel self‑supervision strategy based on view synthesis. Using the same DINOv2 features, a Dense Prediction Transformer (DPT) decoder predicts a monocular depth map for the source image. With known camera intrinsics K and a randomly sampled relative pose Ts→t, each source pixel is back‑projected into 3D, transformed to the target view, and re‑projected to obtain a synthetic target pixel location pgt. By painting the source RGB values onto these projected locations (with z‑buffer handling), a warped novel view Iw is generated, and a dense set of pseudo‑ground‑truth correspondences { (ps, pgt) } is obtained.

These synthetic pairs serve as positive examples in a contrastive triplet loss. For each anchor‑positive pair, a negative is selected as the most distant pixel in feature space within the same image. The loss encourages the adaptation layer Ψ to pull positive descriptors together while pushing negatives apart, effectively reshaping the semantic space into a matching‑friendly embedding space.

After training, the model can directly match any pair of endoscopic images without any external keypoint detector. To further improve geometric precision, the authors apply phase‑correlation based sub‑pixel refinement: each matched patch pair is transformed to the Fourier domain, the cross‑power spectrum is computed, and the peak location yields a fine displacement (Δu, Δv) that is added to the original pixel coordinates. This step reduces epipolar error and enhances downstream tasks such as pose estimation.

The method is evaluated on the SCARED dataset, which contains diverse organ scenes, lighting conditions, and camera motions. Comparisons include classical detectors (SIFT, ORB), recent learning‑based descriptors (HardNet, D2‑Net), and transformer matchers (SuperGlue, LoFTR). The proposed pipeline achieves higher matching precision and lower mean epipolar distance than all baselines. Notably, adding the adaptation transformer improves DINOv2‑only matching by roughly 12 % in precision, while the self‑supervised view‑synthesis supervision contributes to a 30 % reduction in epipolar error. Computationally, the full system runs at approximately 30 FPS on a modern GPU, satisfying real‑time requirements for intra‑operative use.

In summary, the paper makes three principal contributions: (1) a lightweight transformer adaptation module that converts semantic DINOv2 features into dense, matchable descriptors; (2) a novel self‑supervised training scheme that leverages monocular depth estimation and synthetic view generation to obtain pseudo‑ground‑truth correspondences without manual labeling; and (3) an end‑to‑end matching pipeline that combines contrastive learning with Fourier‑based sub‑pixel refinement. These innovations collectively enable high‑quality, real‑time feature matching in the demanding domain of endoscopic surgery, paving the way for more reliable AR overlays, 3D reconstructions, and robotic navigation in the operating room.