Secure AI-Driven Super-Resolution for Real-Time Mixed Reality Applications

Immersive formats such as 360° and 6DoF point cloud videos require high bandwidth and low latency, posing challenges for real-time AR/VR streaming. This work focuses on reducing bandwidth consumption and encryption/decryption delay, two key contributors to overall latency. We design a system that downsamples point cloud content at the origin server and applies partial encryption. At the client, the content is decrypted and upscaled using an ML-based super-resolution model. Our evaluation demonstrates a nearly linear reduction in bandwidth/latency, and encryption/decryption overhead with lower downsampling resolutions, while the super-resolution model effectively reconstructs the original full-resolution point clouds with minimal error and modest inference time.

💡 Research Summary

The paper addresses two critical bottlenecks in immersive mixed‑reality (MR) streaming—high bandwidth consumption and the latency introduced by encryption/decryption—by tightly coupling point‑cloud down‑sampling, attribute‑based encryption (ABE), and an AI‑driven super‑resolution (SR) pipeline. At the origin server, each point‑cloud frame is progressively down‑sampled by removing every second point, yielding four resolution levels (100 % → 50 % → 25 % → 12.5 %). The reduced data are then partially encrypted using ABE; the authors adopt a “full‑coordinate” policy that encrypts X, Y, and Z values, embedding the ciphertext as metadata while discarding the plaintext coordinates. ABE’s policy‑based access control enables fine‑grained, attribute‑driven decryption without per‑user key distribution, thereby lowering cryptographic overhead and preserving CDN caching benefits.

On the client side, the encrypted frames are retrieved from a CDN, decrypted using the user’s attribute‑derived private key, and fed into a super‑resolution module that reconstructs the original high‑resolution point cloud. Rather than employing a deep convolutional network, the authors deliberately choose a Random Forest regression model. This decision is motivated by the limited size of the training set, the need for interpretability, and the desire for low inference latency on commodity hardware. For each sparse point, a feature vector is constructed consisting of its normalized coordinates, the mean distance to its 16 nearest sparse neighbors (computed via a KD‑tree), and a rank indicator (1 or 2) that distinguishes the two nearest dense neighbors used during training. The model, comprising 300 trees with a maximum depth of 24, learns to predict displacement vectors (Δx, Δy, Δz) that map the sparse point to its dense counterparts. Because two ranks are evaluated per input point, the output density is doubled (2× up‑sampling). Higher up‑sampling factors (4×, 8×) can be achieved by iteratively applying the model to its own output.

The experimental evaluation uses two Open3D datasets—LivingRoom (56 scenes) and Office (53 scenes). The authors measure three primary metrics: (1) bandwidth reduction as a function of down‑sampling ratio, (2) ABE encryption/decryption time, and (3) SR reconstruction quality (e.g., Chamfer Distance). Results show an almost linear decrease in both transmitted data size and cryptographic latency as the down‑sampling level increases. The Random Forest SR restores the original geometry with negligible error even at the most aggressive 12.5 % down‑sampling, and inference times stay well below 30 ms per frame, satisfying real‑time MR requirements. The study also compares two training regimes: Model A (trained on one environment, tested on the other) and Model B (trained on both environments). Model B demonstrates better cross‑domain generalization, confirming that the learned geometric priors are transferable across indoor scenes.

Key contributions include: (i) a novel integration of ABE with point‑cloud down‑sampling that simultaneously cuts bandwidth and cryptographic overhead, (ii) the demonstration that a lightweight Random Forest can achieve high‑fidelity SR for volumetric data, and (iii) a comprehensive end‑to‑end pipeline that balances the classic trade‑off triangle of bandwidth, latency, and security. Limitations are acknowledged: the current prototype does not incorporate adaptive bitrate (ABR) mechanisms or DASH streaming, and the Random Forest approach may not scale to very high up‑sampling ratios where deep learning methods typically excel. Future work is outlined to incorporate ABR/DASH, explore deep‑learning‑based SR for higher scaling factors, and develop dynamic, policy‑driven ABE schemes that adapt to network conditions and user contexts, thereby extending the framework toward a production‑grade, QoE‑aware MR streaming system.