arXiv:2412.07744cs.CV2024-12CVPR被引 55

让视频精准匹配指定艺术风格,避免内容泄露且保持时序连贯。

StyleMaster: Stylize Your Video with Artistic Generation and Translation

  • 基于提示-图像块相似性筛选风格特征,保留局部纹理
  • 构建配对风格数据集,提升全局风格一致性至92.3%
  • 轻量级运动适配器桥接图像到视频生成鸿沟

风格控制在视频生成中日益重要。现有方法常生成与给定风格偏差大、存在内容泄露且难以实现风格迁移的问题。我们首次发现风格提取阶段至关重要,现有方法侧重全局风格而忽略局部纹理。为保留纹理特征并防止内容泄露,我们根据提示-图像块相似性过滤内容相关区块,保留风格相关部分;针对全局风格提取,通过模型幻觉生成配对风格数据集以支持对比学习,显著提升绝对风格一致性。此外,为弥合图像到视频的差距,我们在静态图像上训练轻量级运动适配器,隐式增强风格化程度,并使图像训练模型可无缝应用于视频。得益于这些改进,StyleMaster不仅在风格相似度和时序连贯性上显著优于基线,还能通过灰框ControlNet轻松扩展至视频风格迁移。大量实验与可视化表明,该方法能有效生成高质量、符合文本内容且紧密贴近参考图像风格的视频。项目主页见 https://zixuan-ye.github.io/stylemaster

原文摘要 · Abstract (English)

Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at https://zixuan-ye.github.io/stylemaster

视频生成风格迁移扩散模型图像到视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。