Training a Convolutional Neural Network for Appearance-Invariant Place Recognition

Place recognition is one of the most challenging problems in computer vision, and has become a key part in mobile robotics and autonomous driving applications for performing loop closure in visual SLAM systems. Moreover, the difficulty of recognizing a revisited location increases with appearance changes caused, for instance, by weather or illumination variations, which hinders the long-term application of such algorithms in real environments. In this paper we present a convolutional neural network (CNN), trained for the first time with the purpose of recognizing revisited locations under severe appearance changes, which maps images to a low dimensional space where Euclidean distances represent place dissimilarity. In order for the network to learn the desired invariances, we train it with triplets of images selected from datasets which present a challenging variability in visual appearance. The triplets are selected in such way that two samples are from the same location and the third one is taken from a different place. We validate our system through extensive experimentation, where we demonstrate better performance than state-of-art algorithms in a number of popular datasets.

💡 Research Summary

Place recognition is a critical component of visual SLAM systems, enabling loop‑closure detection for mobile robots and autonomous vehicles. Traditional bag‑of‑words (BoW) approaches such as DBoW2 rely on handcrafted local descriptors (ORB, BRIEF) and perform well in static, illumination‑stable environments, but they degrade dramatically under severe appearance changes caused by weather, season, or lighting. Recent works have also explored using generic convolutional neural network (CNN) features extracted from networks pre‑trained for object classification (e.g., CaffeNet/OverFeat), yet these features are not optimized for the specific invariances required in place recognition.

The authors propose a dedicated CNN that learns an embedding space where Euclidean distances directly reflect place dissimilarity, even when the same location is observed under drastically different visual conditions. The core training strategy employs a triplet loss: each training sample consists of a query image (x_i), a “positive” image (x_j) from the same physical location (but possibly captured under a different season, illumination, or viewpoint), and a “negative” image (x_k) from a different location. The loss function (C = \max{0, 1 - |h(x_i)-h(x_k)|^2 + |h(x_i)-h(x_j)|^2}) enforces that the distance between the query and the negative is at least a margin (\beta=1) larger than the distance to the positive. This encourages the network to compress intra‑place variations while expanding inter‑place separations.

To avoid training a deep network from scratch on limited data, the authors fine‑tune the first four convolutional layers of the ImageNet‑pretrained CaffeNet (AlexNet) and replace the remaining fully‑connected layers with a single new fully‑connected layer that outputs a compact descriptor (the exact dimensionality is not specified but is on the order of a few hundred). By discarding the original fully‑connected layers, the input resolution can be reduced to 160 × 120 pixels, dramatically lowering memory usage and computational cost. During fine‑tuning, the pretrained layers are updated with a learning‑rate scaled by 1/1000, while the new layer uses a base learning rate of 0.001. Regularization ((\lambda=5\times10^{-4})) and a margin of 1 are applied, and training proceeds for 40 000 iterations (≈1.2 M triplets).

Triplet construction draws from three complementary datasets that together cover a wide range of appearance variations:

• KITTI – urban driving sequences captured in daylight, providing diverse viewpoints and precise GPS ground truth for selecting positive pairs with varying relative poses and negatives that are truly distinct places.
• Alderley – two 8 km routes recorded at the same location under clear morning light and stormy night conditions, offering extreme illumination and weather changes.
• Nordland – a 728 km train journey recorded in each of the four seasons, delivering the most challenging seasonal appearance shifts.

For each dataset, positives are chosen as images of the same place under different conditions, while negatives are selected from different GPS positions, ensuring that the network experiences both viewpoint and appearance diversity during training.

The evaluation compares the proposed method against two state‑of‑the‑art baselines: (1) DBoW2, an ORB‑based BoW system widely used in SLAM, and (2) generic CNN features obtained from the Conv4 layer of CaffeNet, as suggested in prior work. All methods generate a descriptor for each image; similarity is measured by Euclidean distance and visualized via normalized confusion matrices. Experiments are performed on the three datasets, focusing on scenarios with severe appearance changes.

Results show that the learned CNN consistently outperforms both baselines. On the Nordland dataset, where seasonal variation is extreme, the proposed method achieves up to 18 % higher correct‑match rate than DBoW2 and 12 % higher than generic CaffeNet features. On the Alderley night‑day pair, it gains roughly 15 % improvement over the baselines. Moreover, because the network operates on low‑resolution inputs and lacks heavy fully‑connected layers, inference is fast: about 0.8 ms per frame on a standard CPU, roughly twice as fast as DBoW2 and four times faster than the generic CNN pipeline. Memory consumption is also reduced by more than 30 %.

The paper discusses limitations: the need for accurate GPS or ground‑truth location data to construct triplets, and the fact that the current system does not incorporate sequence‑level post‑processing (e.g., SeqSLAM) which could further boost robustness. Future work is suggested in automatic triplet mining, integration with lightweight architectures such as MobileNet or EfficientNet, and embedding the method directly into real‑time SLAM loops for loop‑closure detection.

In summary, this work demonstrates that a CNN explicitly trained with a triplet loss on carefully curated multi‑condition datasets can learn appearance‑invariant place descriptors that surpass traditional BoW and generic CNN features both in recognition accuracy and computational efficiency. The approach provides a practical pathway toward robust, long‑term visual place recognition for autonomous systems operating in dynamically changing environments.