无需训练即可实现局部视频风格迁移，保持细节和时序一致性
UniVST: A Unified Framework for Training-free Localized Video Style Transfer
- 基于扩散模型的点匹配掩码传播，省去追踪模块
- 无需训练的自适应归一化机制，兼顾内容保真与风格丰富性
- 滑动窗口平滑策略提升时序一致性，减少伪影
本文提出UniVST，一种基于扩散模型的免训练局部视频风格迁移统一框架。该方法不依赖训练，突破了现有扩散模型仅支持全局风格迁移的限制。首先，提出基于DDIM反演特征图的点匹配掩码传播策略，简化架构且无需额外追踪模型。其次，设计免训练的AdaIN引导式局部风格化机制，在潜在空间和注意力层协同工作，平衡内容保真度与风格丰富性，缓解直接视频风格化导致的局部细节丢失问题。最后，引入滑动窗口一致平滑方案，利用像素空间中的光流信息并优化预测噪声以更新潜在表示，显著提升时序一致性并减少伪影。在定量与定性评估中，UniVST均优于现有方法，有效解决主物体风格保留、时序连贯性及细节完整性等挑战。代码已开源：https://github.com/QuanjianSong/UniVST。
原文摘要 · Abstract (English)
This paper presents UniVST, a unified framework for localized video style transfer based on diffusion models. It operates without the need for training, offering a distinct advantage over existing diffusion methods that transfer style across entire videos. The endeavors of this paper comprise: (1) A point-matching mask propagation strategy that leverages the feature maps from the DDIM inversion. This streamlines the model's architecture by obviating the need for tracking models. (2) A training-free AdaIN-guided localized video stylization mechanism that operates at both the latent and attention levels. This balances content fidelity and style richness, mitigating the loss of localized details commonly associated with direct video stylization. (3) A sliding-window consistent smoothing scheme that harnesses optical flow within the pixel representation and refines predicted noise to update the latent space. This significantly enhances temporal consistency and diminishes artifacts in stylized video. Our proposed UniVST has been validated to be superior to existing methods in quantitative and qualitative metrics. It adeptly addresses the challenges of preserving the primary object's style while ensuring temporal consistency and detail preservation. Our code is available at https://github.com/QuanjianSong/UniVST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。