MCTR: Multi Camera Tracking Transformer

Multi-camera tracking plays a pivotal role in various real-world applications. While end-to-end methods have gained significant interest in single-camera tracking, multi-camera tracking remains predominantly reliant on heuristic techniques. In response to this gap, this paper introduces Multi-Camera Tracking tRansformer (MCTR), a novel end-to-end approach tailored for multi-object detection and tracking across multiple cameras with overlapping fields of view. MCTR leverages end-to-end detectors like DEtector TRansformer (DETR) to produce detections and detection embeddings independently for each camera view. The framework maintains set of track embeddings that encaplusate global information about the tracked objects, and updates them at every frame by integrating the local information from the view-specific detection embeddings. The track embeddings are probabilistically associated with detections in every camera view and frame to generate consistent object tracks. The soft probabilistic association facilitates the design of differentiable losses that enable end-to-end training of the entire system. To validate our approach, we conduct experiments on MMPTrack and AI City Challenge, two recently introduced large-scale multi-camera multi-object tracking datasets.

💡 Research Summary

The paper introduces MCTR (Multi‑Camera Tracking Transformer), an end‑to‑end framework that extends the DETR object detector to the multi‑camera multi‑object tracking (MCT) problem. In MCTR each camera stream is processed independently by a vanilla DETR, producing a set of detection embeddings, class scores and bounding boxes. In parallel, a global set of track embeddings is maintained; these embeddings are intended to capture the identity of each object across all cameras and time steps. At every frame the track embeddings are updated by a cross‑attention mechanism that attends to the detection embeddings from every camera. Each camera has its own cross‑attention parameters, allowing the model to learn view‑specific geometric relationships. The outputs of the per‑camera cross‑attention modules are averaged, passed through a self‑attention block and a feed‑forward network, yielding the refreshed track embeddings.

Association between detections and tracks is performed with a scaled dot‑product attention operation that treats detections as queries and tracks as keys/values. This produces, for each camera view v, a probability matrix Aᵥ where Aᵥ(d, t) denotes the likelihood that detection d belongs to track t. Because the operation is fully differentiable, the whole pipeline can be trained jointly.

Training uses three families of loss functions. First, the standard DETR detection loss (classification log‑likelihood, L1 box loss and GIoU loss) is applied independently per camera, together with auxiliary losses from intermediate decoder layers. Second, a cross‑camera track loss enforces identity consistency: for any pair of detections d₁ from view v₁ and d₂ from view v₂, the model computes the probability that they share the same track by summing over tracks P(d₁, d₂)=∑ₜ Aᵥ₁(d₁, t)·Aᵥ₂(d₂, t). Ground‑truth pairwise labels y(d₁, d₂) are derived from the Hungarian matching used for detection loss. A binary cross‑entropy loss is then applied to P(d₁, d₂) only when y is defined. Third, auxiliary track losses encourage diversity and stability of the track embeddings themselves.

The authors evaluate MCTR on two large‑scale multi‑camera benchmarks: MMPTrack and the AI City Challenge. Compared with strong heuristic baselines that combine re‑identification, homography and clustering, MCTR achieves comparable or higher IDF1 and MOTA scores, with a notable reduction in ID switches, especially in scenarios with long occlusions and frequent camera transitions. Qualitative visualizations show that track IDs remain consistent across overlapping views, confirming that the global track embeddings successfully encode cross‑view identity information.

Key contributions of the work are: (1) the introduction of a global track embedding space that is distinct from the per‑camera detection embeddings, enabling unified identity representation across cameras; (2) a fully differentiable probabilistic association mechanism that can be optimized end‑to‑end together with detection; (3) a novel loss formulation that leverages pairwise detection consistency to train the model without relying on a fixed initial matching.

The paper also discusses limitations and future directions. Currently the method assumes substantial overlap between camera fields of view; extending it to non‑overlapping or moving cameras will require additional geometric reasoning or learned view‑invariant embeddings. Incorporating temporal memory (e.g., recurrent networks) into the track embeddings could improve long‑term consistency, while graph‑based constraints might better capture spatial relationships among tracks. Finally, although the experiments focus on 2‑D pedestrian tracking, the architecture is generic and could be adapted to other object categories or to 3‑D multi‑view tracking.

In summary, MCTR demonstrates that transformer‑based end‑to‑end learning, previously successful for single‑camera MOT, can be scaled to the multi‑camera domain by separating local detection features from a global identity space and by training with a probabilistic cross‑camera association loss. The released code and models provide a solid baseline for future research on holistic, data‑driven multi‑camera tracking systems.