用图像数据训练视频风格迁移,保持内容连贯性与风格准确。
PickStyle: Video-to-Video Style Transfer with Context-Style Adapters
- 在扩散模型中插入低秩适配器,实现高效运动风格迁移。
- 合成带相机运动的视频片段,保留时间一致性。
- 分离文本与视频引导方向,确保内容不被风格干扰。
我们研究基于扩散模型的视频风格迁移任务,目标是在保留输入视频上下文的同时,将其渲染为由文本提示指定的目标风格。主要挑战是缺乏成对视频数据进行监督。我们提出PickStyle框架,通过在预训练视频扩散模型的条件模块自注意力层中插入低秩适配器,实现对运动风格迁移的高效专化,同时保持视频内容与风格的良好对齐。为弥合静态图像监督与动态视频之间的差距,我们利用成对图像施加共享增强(模拟相机运动)构建合成训练片段,以保留时间先验。此外,提出上下文-风格无分类器指导(CS-CFG),将无分类器指导分解为独立的文本(风格)与视频(上下文)方向,确保生成视频中上下文信息得以保留,风格有效转移。跨基准实验表明,该方法在时间连贯性、风格忠实度和内容保真度上均优于现有基线,兼具定性与定量优势。
原文摘要 · Abstract (English)
We address the task of video style transfer with diffusion models, where the goal is to preserve the context of an input video while rendering it in a target style specified by a text prompt. A major challenge is the lack of paired video data for supervision. We propose PickStyle, a video-to-video style transfer framework that augments pretrained video diffusion backbones with style adapters and benefits from paired still image data with source-style correspondences for training. PickStyle inserts low-rank adapters into the self-attention layers of conditioning modules, enabling efficient specialization for motion-style transfer while maintaining strong alignment between video content and style. To bridge the gap between static image supervision and dynamic video, we construct synthetic training clips from paired images by applying shared augmentations that simulate camera motion, ensuring temporal priors are preserved. In addition, we introduce Context-Style Classifier-Free Guidance (CS-CFG), a novel factorization of classifier-free guidance into independent text (style) and video (context) directions. CS-CFG ensures that context is preserved in generated video while the style is effectively transferred. Experiments across benchmarks show that our approach achieves temporally coherent, style-faithful, and content-preserving video translations, outperforming existing baselines both qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。