无需成对视频,用强化学习实现少样本视频风格迁移。
Video to Video Generative Adversarial Network for Few-shot Learning Based on Policy Gradient
- 基于策略梯度训练,用ConvLSTM捕捉时空特征。
- 少样本下生成连贯视频,效果优于依赖成对数据的方法。
- 适合无配对数据的视频风格转换场景,如艺术创作、跨域合成。
近年来,深度强化学习与生成对抗网络(GAN)的进步推动了视频到视频合成技术的发展。本文提出RL-V2V-GAN,一种基于强化学习的无监督条件视频到视频合成新方法。该方法在保持源视频风格的同时,学习从源域到目标域的映射关系。通过策略梯度进行训练,并设计细粒度GAN架构,结合时空对抗目标,利用ConvLSTM层捕捉空间与时间信息。对抗损失有助于内容转换同时保留风格。与传统需要成对输入的方法不同,本方法无需成对数据,因此在目标域视频极少(即少样本学习)时尤为有效。实验表明,RL-V2V-GAN能生成时间连贯的视频结果,展现出在视频到视频合成领域的应用潜力。
原文摘要 · Abstract (English)
The development of sophisticated models for video-to-video synthesis has been facilitated by recent advances in deep reinforcement learning and generative adversarial networks (GANs). In this paper, we propose RL-V2V-GAN, a new deep neural network approach based on reinforcement learning for unsupervised conditional video-to-video synthesis. While preserving the unique style of the source video domain, our approach aims to learn a mapping from a source video domain to a target video domain. We train the model using policy gradient and employ ConvLSTM layers to capture the spatial and temporal information by designing a fine-grained GAN architecture and incorporating spatio-temporal adversarial goals. The adversarial losses aid in content translation while preserving style. Unlike traditional video-to-video synthesis methods requiring paired inputs, our proposed approach is more general because it does not require paired inputs. Thus, when dealing with limited videos in the target domain, i.e., few-shot learning, it is particularly effective. Our experiments show that RL-V2V-GAN can produce temporally coherent video results. These results highlight the potential of our approach for further advances in video-to-video synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。