arXiv:2504.05537cs.CVcs.AI2025-04被引 3

用生成时间序列模型实现低延迟视频动作迁移,节省带宽。

Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling

  • 用关键点建模动作,通过VRNN和GRU-NF预测未来帧。
  • VRNN在多步预测中误差最低,适合稳定场景;GRU-NF生成更多样化结果。
  • 适用于视频会议、远程医疗等实时低带宽应用,支持可控生成。

动作迁移技术通过将驱动视频中的运动动态转移到源图像上合成新视频。本文提出一种基于深度学习的框架,实现实时视频动作迁移,对视频会议、远程健康监测、虚拟现实交互和视觉异常检测等带宽敏感应用至关重要。该方法使用关键点作为时序运动的语义紧凑表示。为降低传输带宽,采用两种生成时间序列模型——VRNN与GRU-NF——对关键点进行预测。预测的关键点通过基于光流的模块与生成网络结合,还原为真实视频帧,从而实现高效、低帧率视频传输。根据应用场景,该框架可生成确定性未来序列或采样多样化的可能未来。实验表明,VRNN在需要稳定高精度多步预测的应用中表现最优(最低MAE),尤其在高不确定性、多模态场景下具有优势。其通过递归条件化的随机隐变量捕捉历史上下文与时间变化。而GRU-NF则在保持高视觉质量的同时生成更丰富的多样性视频,得益于其在关键点与其隐表示之间学习到的可逆精确似然映射,支持丰富且可控的多样化关键点序列采样。本工作为下一代需实时、低带宽、语义可控视频生成的AI系统奠定基础。

原文摘要 · Abstract (English)

Motion Transfer is a technique that synthesizes videos by transferring motion dynamics from a driving video to a source image. In this work we propose a deep learning-based framework to enable real-time video motion transfer which is critical for enabling bandwidth-efficient applications such as video conferencing, remote health monitoring, virtual reality interaction, and vision-based anomaly detection. This is done using keypoints which serve as semantically meaningful, compact representations of motion across time. To enable bandwidth savings during video transmission we perform forecasting of keypoints using two generative time series models VRNN and GRU-NF. The predicted keypoints are transformed into realistic video frames using an optical flow-based module paired with a generator network, thereby enabling efficient, low-frame-rate video transmission. Based on the application this allows the framework to either generate a deterministic future sequence or sample a diverse set of plausible futures. Experimental results demonstrate that VRNN achieves the best point-forecast fidelity (lowest MAE) in applications requiring stable and accurate multi-step forecasting and is particularly competitive in higher-uncertainty, multi-modal settings. This is achieved by introducing recurrently conditioned stochastic latent variables that carry past contexts to capture uncertainty and temporal variation. On the other hand the GRU-NF model enables richer diversity of generated videos while maintaining high visual quality. This is realized by learning an invertible, exact-likelihood mapping between the keypoints and their latent representations which supports rich and controllable sampling of diverse yet coherent keypoint sequences. Our work lays the foundation for next-generation AI systems that require real-time, bandwidth-efficient, and semantically controllable video generation.

动作迁移生成模型实时视频时间序列

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。