A Distributed Framework for Privacy-Enhanced Vision Transformers on the Edge

Nowadays, visual intelligence tools have become ubiquitous, offering all kinds of convenience and possibilities. However, these tools have high computational requirements that exceed the capabilities of resource-constrained mobile and wearable devices. While offloading visual data to the cloud is a common solution, it introduces significant privacy vulnerabilities during transmission and server-side computation. To address this, we propose a novel distributed, hierarchical offloading framework for Vision Transformers (ViTs) that addresses these privacy challenges by design. Our approach uses a local trusted edge device, such as a mobile phone or an Nvidia Jetson, as the edge orchestrator. This orchestrator partitions the user’s visual data into smaller portions and distributes them across multiple independent cloud servers. By design, no single external server possesses the complete image, preventing comprehensive data reconstruction. The final data merging and aggregation computation occurs exclusively on the user’s trusted edge device. We apply our framework to the Segment Anything Model (SAM) as a practical case study, which demonstrates that our method substantially enhances content privacy over traditional cloud-based approaches. Evaluations show our framework maintains near-baseline segmentation performance while substantially reducing the risk of content reconstruction and user data exposure. Our framework provides a scalable, privacy-preserving solution for vision tasks in the edge-cloud continuum.

💡 Research Summary

This paper addresses the critical privacy vulnerabilities inherent in deploying high-performance Vision Transformer (ViT) models for visual intelligence tasks on resource-constrained mobile and wearable devices. While offloading computation to the cloud is a common necessity, it exposes users’ sensitive egocentric visual data (e.g., from AR glasses) to potential reconstruction and misuse during transmission and server-side processing. To mitigate this, the authors propose a novel distributed, hierarchical offloading framework that enhances privacy by design, leveraging the inherent architectural properties of modern ViTs.

The core innovation stems from the observation that state-of-the-art ViTs, such as Swin Transformers or the image encoder in the Segment Anything Model (SAM), utilize a hybrid attention mechanism. They process high-resolution images primarily through localized “window attention” layers, where self-attention is computed within non-overlapping windows of image patches. Cross-window information exchange is handled by a much smaller number of “global attention” layers. The proposed framework repurposes this computational structure for privacy preservation.

The system architecture consists of three tiers: a thin client (e.g., AR glasses), a trusted edge orchestrator (e.g., a smartphone or NVIDIA Jetson device), and multiple independent cloud servers. The trusted orchestrator partitions the input image into a grid of non-overlapping windows. Instead of sending the entire image to a single cloud server, it distributes the computation-intensive window attention layer processing for each window to a different cloud server. Crucially, no single external server receives the complete image; each only processes a small fraction (e.g., 1/25th) of the visual data. The servers, assumed to be honest-but-curious and non-colluding, return the computed window embeddings to the orchestrator. Subsequently, the orchestrator aggregates these partial results and executes the lightweight global attention layers locally within the trusted execution environment to produce the final, comprehensive image embedding. This embedding is then used for downstream vision tasks like segmentation with minimal local computation.

The authors implement and evaluate their framework using the Segment Anything Model (SAM) as a case study, deploying a prototype with Docker and gRPC for edge-cloud communication. Comprehensive evaluations demonstrate the framework’s dual efficacy. First, it maintains near-baseline utility: the segmentation accuracy achieved using embeddings from the privacy-enhanced pipeline is comparable to that of the standard, non-private cloud offloading approach. Second, it significantly enhances privacy. Even under strong adversarial scenarios employing state-of-the-art image reconstruction models (e.g., diffusion models), an attacker with access to only the data processed by a single cloud server cannot meaningfully reconstruct the original scene or identify sensitive objects at a pixel or semantic level. The distributed partitioning inherently limits the information leakage to any single untrusted party.

In summary, this work presents a practical and scalable system-level solution for privacy-preserving visual intelligence on the edge-cloud continuum. It distinguishes itself by providing substantial privacy gains without modifying the underlying AI model architecture or incurring the prohibitive overheads associated with cryptographic techniques like homomorphic encryption. The framework successfully transforms an efficiency-oriented feature of ViTs—windowed attention—into a powerful tool for data minimization and privacy protection.