Privacy-Aware Sharing of Raw Spatial Sensor Data for Cooperative Perception

Cooperative perception between vehicles is poised to offer robust and reliable scene understanding. Recently, we are witnessing experimental systems research building testbeds that share raw spatial sensor data for cooperative perception. While there has been a marked improvement in accuracies and is the natural way forward, we take a moment to consider the problems with such an approach for eventual adoption by automakers. In this paper, we first argue that new forms of privacy concerns arise and discourage stakeholders to share raw sensor data. Next, we present SHARP, a research framework to minimize privacy leakage and drive stakeholders towards the ambitious goal of raw data based cooperative perception. Finally, we discuss open questions for networked systems, mobile computing, perception researchers, industry and government in realizing our proposed framework.

💡 Research Summary

The paper addresses the emerging need for sharing raw spatial sensor data (camera images, lidar point clouds, radar I/Q streams) among connected vehicles to achieve higher‑fidelity cooperative perception. While raw data can dramatically improve detection of small or occluded objects—especially under adverse weather—its adoption faces two critical privacy‑related barriers that have been largely overlooked in recent research.

First, vehicle location privacy is at risk. Current V2X standards already transmit global GPS coordinates and vehicle pose under pseudonyms, but raw visual and point‑cloud data themselves contain enough information for visual localization, sensor‑noise fingerprinting, and driving‑pattern analysis to reconstruct a vehicle’s trajectory over long periods. Modern foundation models such as VGGT and Dust3R can extract accurate 6‑DoF poses from a few frames, making location inference trivial for a malicious receiver.

Second, raw data expose intellectual property (IP) of sensor hardware. Differences in camera stabilization, lidar sampling rates, radar antenna counts, and other low‑level characteristics can be reverse‑engineered from raw streams, threatening the billions of dollars invested by automakers in sensor R&D. Because the automotive supply chain is tiered (Tier‑1, Tier‑2, Tier‑3), most OEMs only receive processed data; sharing raw data would break this protective barrier.

The authors argue that conventional cryptographic approaches—secure multi‑party computation (SMPC) or fully homomorphic encryption (FHE)—are infeasible for real‑time perception, where end‑to‑end latencies must stay within 10‑100 ms and data volumes are massive (megapixels per frame, millions of lidar points). To overcome these challenges, they propose SHARP (Sharing Raw spatial sensor data Privately), a two‑pronged framework.

1. Location‑Obfuscation via Random View Synthesis
Each vehicle locally runs a fast 3D reconstruction using feed‑forward models (e.g., VGGT, Dust3R) that can convert 2‑D images into dense point clouds in milliseconds. Instead of transmitting the true sensor viewpoint, the vehicle selects a random virtual viewpoint, renders a synthetic raw sensor stream (image, point cloud, radar histogram) from that viewpoint, and shares the synthetic data. This “novel view synthesis” masks the true pose while preserving the statistical properties needed for downstream perception. A large‑scale simulation on the OP V2V dataset (73 scenes, 50 rollouts each) shows that with a 12 m offset (approximately four lane widths) the confusion rate for identifying the true vehicle rises to 25 % and RMSE exceeds 45 m, indicating substantial privacy gain. The authors discuss practical hurdles: ensuring rendered views are photorealistic enough for downstream models, handling occlusions, and maintaining the required 10 Hz+ update rate.

2. IP‑Leakage Mitigation through an Open Low‑Level Stack
The paper proposes a dual‑stack architecture. An “open stack” implements a hardware‑agnostic, publicly documented signal‑processing pipeline (denoising, basic calibration, conversion to a standardized raw format) that runs on the sensor output and produces shareable data without exposing proprietary sensor characteristics. A “proprietary stack” continues to run locally for high‑performance perception tasks. Vehicles periodically switch between the two stacks (e.g., open stack at 5 Hz, proprietary at 20 Hz). The scheduler must balance compute load, latency introduced by stack swaps, and the cooperative perception reaction time budget. By exposing only a subset of sensor metadata (e.g., a single antenna’s data instead of the full array), manufacturers can protect hardware IP while still providing useful raw data.

The authors acknowledge that standardizing a hardware‑agnostic raw format and open processing blocks is non‑trivial, given the diversity of sensor designs. Nevertheless, they argue that a community‑driven open representation—similar to how image codecs became standardized—could enable cross‑OEM collaboration without sacrificing competitive advantage.

Finally, the paper outlines open research questions: (a) how to select random viewpoints that remain useful for perception while maximizing privacy, (b) how to handle multi‑vehicle synchronization and bandwidth constraints when many cars simultaneously stream synthetic raw data, (c) how to integrate SHARP with existing V2X security primitives (pseudonym certificates, message authentication), and (d) what regulatory or industry‑wide standards are needed to formalize the open stack and privacy‑preserving protocols.

In summary, SHARP offers a concrete, technically grounded pathway to reconcile the performance benefits of raw sensor sharing with the privacy and IP concerns that currently hinder large‑scale deployment. By leveraging recent advances in fast 3D understanding and by proposing a pragmatic dual‑stack architecture, the work charts a realistic roadmap for future cooperative perception systems that can be adopted by automakers, policymakers, and standards bodies alike.