Model Optimization for Multi-Camera 3D Detection and Tracking

Outside-in multi-camera perception is increasingly important in indoor environments, where networks of static cameras must support multi-target tracking under occlusion and heterogeneous viewpoints. We evaluate Sparse4D, a query-based spatiotemporal 3D detection and tracking framework that fuses multi-view features in a shared world frame and propagates sparse object queries via instance memory. We study reduced input frame rates, post-training quantization (INT8 and FP8), transfer to the WILDTRACK benchmark, and Transformer Engine mixed-precision fine-tuning. To better capture identity stability, we report Average Track Duration (AvgTrackDur), which measures identity persistence in seconds. Sparse4D remains stable under moderate FPS reductions, but below 2 FPS, identity association collapses even when detections are stable. Selective quantization of the backbone and neck offers the best speed-accuracy trade-off, while attention-related modules are consistently sensitive to low precision. On WILDTRACK, low-FPS pretraining yields large zero-shot gains over the base checkpoint, while small-scale fine-tuning provides limited additional benefit. Transformer Engine mixed precision reduces latency and improves camera scalability, but can destabilize identity propagation, motivating stability-aware validation.

💡 Research Summary

This paper conducts a comprehensive, deployment‑oriented evaluation of Sparse4D, a query‑based spatiotemporal 3D detection and multi‑camera tracking framework, under four practical constraints: reduced inference frame rate, post‑training quantization (PTQ), transfer to the WILDTRACK benchmark, and mixed‑precision fine‑tuning with NVIDIA’s Transformer Engine (TE). The authors introduce a new identity‑stability metric, Average Track Duration (AvgTrackDur), which measures the average continuous time that a tracked identity remains correctly associated, expressed in seconds.

Low‑FPS robustness – Using the AI City 2025 warehouse dataset (native 30 FPS), the authors simulate lower frame rates by skipping frames. They evaluate standard tracking metrics (HOT A, Det A, Ass A, Loc A) together with AvgTrackDur. Results show that Sparse4D tolerates moderate reductions down to about 6 FPS with only minor drops in detection and association scores, and AvgTrackDur remains above 2.5 s. Below 2 FPS, however, the spatiotemporal transformer can no longer propagate queries fast enough to bridge the larger inter‑frame displacements, causing a sharp collapse of identity association (AvgTrackDur falls below 1 s). This confirms that motion‑only association methods (e.g., Kalman filters) are insufficient in very low‑frame regimes.

Post‑training quantization – Two PTQ pipelines are explored. INT8 quantization is built on an A100 GPU using TensorRT; FP8 quantization is built on H100/H200 GPUs using NVIDIA ModelOpt and a custom FP8‑friendly ONNX conversion. The authors sweep quantization scope via regex targeting: (i) backbone only, (ii) backbone + neck, and (iii) backbone + neck + attention. Selective INT8 quantization of backbone and neck yields the best speed‑accuracy trade‑off: inference speed improves by ~1.8× while AvgTrackDur and Ass A degrade by less than 3 %. Extending INT8 to attention layers introduces severe activation outliers, causing AvgTrackDur to drop by >30 % and destabilizing the memory‑based query updates. FP8 shows similar patterns: backbone + neck quantization reduces latency further, but attention quantization again harms identity stability. The study concludes that attention modules are highly sensitive to low‑precision arithmetic and should remain in higher precision (FP16/FP32).

Transfer to WILDTRACK – WILDTRACK is an outdoor multi‑camera dataset captured at 2 FPS, with different camera geometry, lighting, and pedestrian density. The authors build a full adaptation pipeline: (a) convert the original grid‑based position‑ID annotations to a TAO‑compatible format with 3‑D bounding boxes, (b) regenerate a 3‑D anchor bank by k‑means clustering over ground‑truth centers (K = 900), and (c) adjust loss functions to focus on pedestrian center regression (yaw and size terms disabled). Two pre‑training checkpoints are evaluated: the original AI City checkpoint and a “COSMOS‑augmented” checkpoint trained on synthetic data with a multi‑FPS curriculum (1–30 FPS) and appearance style augmentation. Zero‑shot transfer from the COSMOS checkpoint yields a substantial gain in AvgTrackDur (from 1.9 s to 3.2 s) and modest improvements in detection scores. Fine‑tuning on the limited WILDTRACK data (≈10 epochs, reduced LR = 1e‑5) provides only marginal gains in Det A and Loc A, indicating that the model already captures most of the necessary visual cues after low‑FPS pre‑training.

Mixed‑precision fine‑tuning with Transformer Engine – To assess whether the model can adapt to low‑precision environments rather than being merely post‑quantized, the authors integrate TE into the training loop. They target the first single‑frame decoder layer and the anchor encoder for FP8 compute while keeping the rest of the network in FP16. After 22,500 training steps (~0.5 epoch), inference latency drops by ~28 % and the system scales more gracefully when the number of cameras is increased from 12 to 16 (FPS gain ≈ 1.2×). However, AvgTrackDur declines from 0.8 s to 0.5 s, revealing that FP8 arithmetic introduces subtle numerical noise in the attention‑based query updates, which harms long‑term identity propagation. The authors recommend monitoring identity‑stability metrics during mixed‑precision training and, if necessary, keeping attention layers in higher precision.

Key takeaways

1. Sparse4D remains robust to moderate FPS reductions but fails below ~2 FPS, highlighting the need for stronger motion modeling or higher‑frequency sampling in bandwidth‑constrained deployments.
2. Selective INT8/PTQ of backbone + neck offers the best latency‑accuracy trade‑off; attention modules should stay in FP16/FP32.
3. Low‑FPS pre‑training (COSMOS curriculum) provides large zero‑shot gains when transferring to low‑frame‑rate datasets like WILDTRACK, while extensive fine‑tuning on the target domain yields diminishing returns.
4. Mixed‑precision training with TE can further reduce latency and improve scalability, but may destabilize identity tracking; a hybrid precision scheme (FP8 for early layers, FP16 for attention) is advisable.

Overall, the paper delivers actionable guidelines for deploying query‑based multi‑camera 3D detection and tracking systems in real‑world indoor/outdoor settings, balancing accuracy, identity stability, latency, and training efficiency.