Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling

Motion Transfer is a technique that synthesizes videos by transferring motion dynamics from a driving video to a source image. In this work we propose a deep learning-based framework to enable real-time video motion transfer which is critical for enabling bandwidth-efficient applications such as video conferencing, remote health monitoring, virtual reality interaction, and vision-based anomaly detection. This is done using keypoints which serve as semantically meaningful, compact representations of motion across time. To enable bandwidth savings during video transmission we perform forecasting of keypoints using two generative time series models VRNN and GRU-NF. The predicted keypoints are transformed into realistic video frames using an optical flow-based module paired with a generator network, thereby enabling efficient, low-frame-rate video transmission. Based on the application this allows the framework to either generate a deterministic future sequence or sample a diverse set of plausible futures. Experimental results demonstrate that VRNN achieves the best point-forecast fidelity (lowest MAE) in applications requiring stable and accurate multi-step forecasting and is particularly competitive in higher-uncertainty, multi-modal settings. This is achieved by introducing recurrently conditioned stochastic latent variables that carry past contexts to capture uncertainty and temporal variation. On the other hand the GRU-NF model enables richer diversity of generated videos while maintaining high visual quality. This is realized by learning an invertible, exact-likelihood mapping between the keypoints and their latent representations which supports rich and controllable sampling of diverse yet coherent keypoint sequences. Our work lays the foundation for next-generation AI systems that require real-time, bandwidth-efficient, and semantically controllable video generation.

💡 Research Summary

This paper presents a novel deep learning framework designed to achieve efficient real-time video motion transfer, with a primary focus on drastic bandwidth reduction for applications like video conferencing, telehealth, VR, and anomaly detection. The core innovation lies in integrating generative time-series forecasting into a keypoint-based motion transfer pipeline, enabling both spatial and temporal compression.

The methodology operates through a two-stage process. First, motion is represented not by raw pixels but by a sparse set of “keypoints,” which are semantically meaningful points (e.g., facial landmarks, object corners) extracted from each video frame via a self-supervised detector. These keypoints form a compact, low-dimensional time series representing the motion dynamics. To achieve temporal compression and bandwidth savings, the framework predicts future keypoint sequences instead of transmitting them for every frame. For this prediction task, the authors rigorously investigate and compare two advanced generative time-series models: a Variational Recurrent Neural Network (VRNN) and a Gated Recurrent Unit-Normalizing Flow model (GRU-NF).

The VRNN leverages recurrently conditioned stochastic latent variables to capture uncertainty and temporal dependencies from past context, making it particularly strong for stable and accurate multi-step “point forecasting.” This mode is ideal for applications requiring a single, reliable future prediction, such as video calls or patient monitoring. In contrast, the GRU-NF model combines a GRU’s temporal modeling with the invertible, exact-likelihood mapping of a Normalizing Flow. This allows it to learn a rich latent distribution of future keypoints, enabling the sampling of multiple diverse yet coherent future sequences from the same past observations. This “diverse forecasting” mode is invaluable for scenarios like anomaly detection in manufacturing, where exploring various plausible futures can help identify potential defects.

In the second stage, the predicted future keypoints (either the single VRNN prediction or one of many GRU-NF samples) are fed into an optical flow-based generator network. This module, inspired by architectures like the First Order Motion Model (FOMM), warps a source image according to the motion implied by the keypoints, synthesizing realistic and high-quality video frames corresponding to the forecasted motion.

Comprehensive experiments on multiple benchmark video datasets demonstrate the framework’s effectiveness. The VRNN achieves the lowest Mean Absolute Error (MAE) for point forecasts, confirming its superiority in accuracy-critical settings. The GRU-NF generates samples with higher diversity (measured by metrics like LPIPS and FVD diversity) while maintaining competitive visual quality, proving its strength for multi-modal prediction tasks. The overall system is shown to provide up to 20x bandwidth savings compared to full video transmission, a significant improvement over the ~10x savings from non-predictive keypoint transfer methods.

In conclusion, this work makes a substantial contribution by moving beyond mere spatial compression to achieve joint spatio-temporal efficiency. It provides a flexible framework where the choice of forecasting model (VRNN for fidelity, GRU-NF for diversity) can be tailored to the specific needs of the end application. The research lays a foundational blueprint for next-generation, bandwidth-aware AI systems that require real-time, efficient, and semantically controllable video generation and communication.