mmCooper: A Multi-agent Multi-stage Communication-efficient and Collaboration-robust Cooperative Perception Framework

Collaborative perception significantly enhances individual vehicle perception performance through the exchange of sensory information among agents. However, real-world deployment faces challenges due to bandwidth constraints and inevitable calibration errors during information exchange. To address these issues, we propose mmCooper, a novel multi-agent, multi-stage, communication-efficient, and collaboration-robust cooperative perception framework. Our framework leverages a multi-stage collaboration strategy that dynamically and adaptively balances intermediate- and late-stage information to share among agents, enhancing perceptual performance while maintaining communication efficiency. To support robust collaboration despite potential misalignments and calibration errors, our framework prevents misleading low-confidence sensing information from transmission and refines the received detection results from collaborators to improve accuracy. The extensive evaluation results on both real-world and simulated datasets demonstrate the effectiveness of the mmCooper framework and its components.

💡 Research Summary

The paper introduces mmCooper, a novel cooperative perception framework designed for multi‑agent autonomous driving scenarios where both communication bandwidth and sensor calibration errors pose serious challenges. Unlike prior works that rely on a single fusion stage—early (raw point clouds), intermediate (feature maps), or late (bounding boxes)—mmCooper adopts a dynamic, multi‑stage collaboration strategy that selectively shares information at the intermediate or late stage based on per‑location detection confidence.

The pipeline begins with each vehicle encoding its LiDAR point cloud into a Bird’s Eye View (BEV) feature map using a shared PointPillar encoder. A Confidence‑based Filter Generation (CFG) module then produces two confidence maps: one for intermediate‑stage transmission and another for late‑stage transmission. After Gaussian smoothing, the top‑p % of locations are kept, and a Gumbel‑Softmax sampler determines, in a differentiable manner, whether each location should (a) transmit its intermediate feature, (b) transmit a coarse bounding box, or (c) suppress transmission altogether. This confidence‑driven filtering dramatically reduces the amount of data exchanged while preserving critical information.

For the intermediate stage, the authors propose a Multi‑scale Offset‑aware Fusion (MOF) module. MOF extracts multi‑scale representations (e.g., 1/4, 1/8, 1/16 resolutions) of both ego and collaborator features and applies cross‑attention to fuse target‑region features together with those from neighboring regions. By incorporating spatial context at multiple scales, MOF mitigates misalignment caused by pose noise, communication delay, or calibration drift, yielding robust fused features even under imperfect synchronization.

In the late stage, a Bounding Box Filtering & Calibration (BFC) module leverages the rich intermediate features from the ego vehicle to (i) discard low‑confidence or erroneous boxes received from collaborators, and (ii) refine the remaining boxes’ positions and dimensions. The calibrated collaborator boxes are then merged with the ego’s own detections to produce the final perception output.

Extensive experiments on three benchmarks—OPV2V (real‑world), DAIR‑V2X, and V2XSet (simulated)—demonstrate that mmCooper outperforms the second‑best single‑stage methods by 7.29 %, 1.31 %, and 2.09 % in AP@0.7 respectively. Remarkably, the communication volume required by mmCooper is reduced to merely 1/9 153, 1/156, and 1/18 305 of that used by comparable state‑of‑the‑art approaches. Additional ablation studies show consistent gains across varying bandwidth limits and pose‑error magnitudes, confirming the framework’s robustness.

In summary, mmCooper achieves a three‑fold advancement: (1) a confidence‑driven, adaptive multi‑stage transmission policy that balances perception quality and bandwidth usage; (2) a multi‑scale offset‑aware fusion mechanism that is resilient to calibration errors; and (3) a late‑stage bounding‑box refinement process that capitalizes on intermediate features for accurate final detections. These contributions collectively push cooperative perception closer to practical deployment in bandwidth‑constrained V2X networks.