arXiv:2508.14483cs.CV2025-08中稿 · ICLR被引 8

用文本到视频模型蒸馏概念,修复视频时保真度和连贯性更佳。

Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration

  • 用预训练T2V生成带文本概念的训练数据,蒸馏其语义理解能力。
  • 新控制架构过滤退化特征,提升生成时纹理真实感与时间一致性。
  • 适合需要高保真视频修复的科研与影视制作人员使用。

我们提出Vivid-VR,一种基于DiT的生成式视频修复方法,依托先进的文本到视频(T2V)基础模型,并利用ControlNet控制生成过程以保证内容一致性。然而,传统微调常因多模态对齐不充分导致分布漂移,损害纹理真实性和时间连贯性。为此,我们提出概念蒸馏训练策略,利用预训练T2V模型合成包含文本概念的训练样本,从而将其中的概念理解能力迁移到修复模型中。为增强可控性,我们重新设计控制架构:1)引入控制特征投影器,从输入视频潜在表示中滤除退化伪影,减少其在生成流程中的传播;2)采用双分支结构的ControlNet连接器,结合MLP特征映射与交叉注意力机制,实现动态控制特征获取,兼顾内容保持与自适应控制信号调节。大量实验表明,Vivid-VR在合成与真实世界基准以及AIGC视频上均优于现有方法,显著提升纹理真实感、视觉生动性与时间一致性。代码与模型权重已公开于https://github.com/csbhr/Vivid-VR。

原文摘要 · Abstract (English)

We present Vivid-VR, a DiT-based generative video restoration method built upon an advanced T2V foundation model, where ControlNet is leveraged to control the generation process, ensuring content consistency. However, conventional fine-tuning of such controllable pipelines frequently suffers from distribution drift due to limitations in imperfect multimodal alignment, resulting in compromised texture realism and temporal coherence. To tackle this challenge, we propose a concept distillation training strategy that utilizes the pretrained T2V model to synthesize training samples with embedded textual concepts, thereby distilling its conceptual understanding to preserve texture and temporal quality. To enhance generation controllability, we redesign the control architecture with two key components: 1) a control feature projector that filters degradation artifacts from input video latents to minimize their propagation through the generation pipeline, and 2) a new ControlNet connector employing a dual-branch design. This connector synergistically combines MLP-based feature mapping with cross-attention mechanism for dynamic control feature retrieval, enabling both content preservation and adaptive control signal modulation. Extensive experiments show that Vivid-VR performs favorably against existing approaches on both synthetic and real-world benchmarks, as well as AIGC videos, achieving impressive texture realism, visual vividness, and temporal consistency. The codes and checkpoints are publicly available at https://github.com/csbhr/Vivid-VR.

视频修复扩散模型概念蒸馏ControlNet

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。