GenAI-enabled Residual Motion Estimation for Energy-Efficient Semantic Video Communication

Semantic communication addresses the limitations of the Shannon paradigm by focusing on transmitting meaning rather than exact representations, thereby reducing unnecessary resource consumption. This is particularly beneficial for video, which dominates network traffic and demands high bandwidth and power, making semantic approaches ideal for conserving resources while maintaining quality. In this paper, we propose a Predictability-aware and Entropy-adaptive Neural Motion Estimation (PENME) method to address challenges related to high latency, high bitrate, and power consumption in video transmission. PENME makes per-frame decisions to select a residual motion extraction model, convolutional neural network, vision transformer, or optical flow, using a five-step policy based on motion strength, global motion consistency, peak sharpness, heterogeneity, and residual error. The residual motions are then transmitted to the receiver, where the frames are reconstructed via motion-compensated updates. Next, a selective diffusion-based refinement, the Latent Consistency Model (LCM-4), is applied on frames that trigger refinement due to low predictability or large residuals, while predictable frames skip refinement. PENME also allocates radio resource blocks with awareness of residual motion and channel state, reducing power consumption and bandwidth usage while maintaining high semantic similarity. Our simulation results on the Vimeo90K dataset demonstrate that the proposed PENME method handles various types of video, outperforming traditional communication, hybrid, and adaptive bitrate semantic communication techniques, achieving 40% lower latency, 90% less transmitted data, and 35% higher throughput. For semantic communication metrics, PENME improves PSNR by about 40%, increases MS-SSIM by roughly 19%, and reduces LPIPS by nearly 35%, compared with the baseline methods.

💡 Research Summary

The paper addresses the growing demand for efficient video transmission in beyond‑5G/6G networks by moving from traditional Shannon‑style bit‑accurate communication to a semantic‑driven approach that transmits only the meaning‑relevant information. The authors propose a novel framework called Predictability‑aware and Entropy‑adaptive Neural Motion Estimation (PEN ME). PEN ME operates on a per‑frame basis and extracts five normalized motion signals from each consecutive frame pair: motion strength, global shift consistency, peak sharpness, local heterogeneity, and residual error after global compensation. These signals are combined into a score that determines which of three motion‑extraction models—optical flow, a vision transformer (ViT), or a lightweight convolutional neural network (CNN)—should be used for that segment. Strong, coherent motion triggers an optical‑flow estimator; highly irregular or complex motion invokes the ViT; weak, homogeneous motion is handled by the CNN. By dynamically selecting the most appropriate estimator, the system balances computational load and motion‑estimation accuracy.

Only the residual motion vectors produced by the chosen estimator are entropy‑coded and transmitted. At the receiver, a motion‑compensated update reconstructs a coarse frame. If the frame is deemed unpredictable (high entropy) or the residual magnitude exceeds a preset threshold, a conditional diffusion module called Latent Consistency Model‑4 (LCM‑4) is applied. LCM‑4 performs a few (typically 3‑5) latent‑space diffusion steps, projecting noisy or incomplete motion vectors onto a temporally coherent manifold. This selective, lightweight diffusion refinement dramatically improves visual fidelity while avoiding the heavy computational burden of full‑scale diffusion decoders.

In addition to motion handling, PEN ME incorporates channel‑aware radio resource block (RB) allocation. The system jointly considers the residual motion complexity and the current channel state information (CSI) to assign just enough RBs and transmission power for each frame. Frames with low residuals and good channel conditions receive minimal resources, whereas frames with high residuals or poor CSI are allocated extra RBs and power. This adaptive allocation reduces overall power consumption and improves throughput.

The authors evaluate PEN ME on the Vimeo90K dataset under a variety of scenarios, including static scenes, rapid camera pans, complex object motion, and low‑SNR wireless channels. Compared with adaptive‑bitrate video semantic communication (ABR VSC), hybrid semantic codecs, and conventional video transmission, PEN ME achieves a 40 % reduction in end‑to‑end latency, a 90 % decrease in transmitted data volume, and a 35 % increase in throughput. Quality metrics improve substantially: peak signal‑to‑noise ratio (PSNR) rises by roughly 40 %, multi‑scale structural similarity (MS‑SSIM) by about 19 %, and learned perceptual image patch similarity (LPIPS) drops by nearly 35 %. The ViT‑plus‑LCM‑4 combination is especially beneficial for high‑complexity motion, while the CNN path keeps computation low for simple motion, enabling real‑time operation.

Key contributions of the work are: (1) a dynamic, entropy‑aware motion‑estimation selector that intelligently switches among CNN, optical flow, and ViT; (2) a conditional, few‑step diffusion refinement module that offers high perceptual quality with modest compute; (3) an edge‑assisted design that offloads heavy processing from the transmitter and keeps receiver load manageable; and (4) a comprehensive experimental validation demonstrating superior latency, bandwidth efficiency, and visual quality across diverse channel conditions. The paper concludes with suggestions for future research, including online learning for on‑the‑fly adaptation, multi‑user MIMO extensions, and integration with task‑oriented semantic objectives.