用语义蒸馏提升视频超分细节一致性，解决扩散模型帧间不连贯问题
SeeClear: Semantic Distillation Enhances Pixel Condensation for Video Super-Resolution
- 引入语义蒸馏与像素凝聚双模块，动态对齐跨帧语义
- 在Reds4、Shakir、Vimeo-910上实现峰值指标超越现有方法
- 适合关注视频超分辨率与生成一致性研究的开发者
基于扩散的视频超分辨率（VSR）虽能生成视觉逼真的视频，但因随机扰动导致帧间细节不一致。传统像素级对齐在扩散处理中失效。为此，本文提出SeeClear框架，采用以实例为中心和通道级语义控制的条件视频生成机制。该框架融合语义蒸馏器与像素凝聚器，从低分辨率帧中提取并上采样语义细节。实例中心对齐模块（InCAM）利用视频片段级标记动态关联帧内及跨帧像素，增强时序连贯性。通道级纹理聚合记忆（CaTeGory）引入外部语义纹理知识，强化长期特征表达。同时，通过ResShift机制改进模糊扩散过程，在锐度与扩散效果间取得平衡。大量实验表明，该框架优于当前最先进的扩散型VSR方法。代码已开源：https://github.com/Tang1705/SeeClear-NeurIPS24。
原文摘要 · Abstract (English)
Diffusion-based Video Super-Resolution (VSR) is renowned for generating perceptually realistic videos, yet it grapples with maintaining detail consistency across frames due to stochastic fluctuations. The traditional approach of pixel-level alignment is ineffective for diffusion-processed frames because of iterative disruptions. To overcome this, we introduce SeeClear--a novel VSR framework leveraging conditional video generation, orchestrated by instance-centric and channel-wise semantic controls. This framework integrates a Semantic Distiller and a Pixel Condenser, which synergize to extract and upscale semantic details from low-resolution frames. The Instance-Centric Alignment Module (InCAM) utilizes video-clip-wise tokens to dynamically relate pixels within and across frames, enhancing coherency. Additionally, the Channel-wise Texture Aggregation Memory (CaTeGory) infuses extrinsic knowledge, capitalizing on long-standing semantic textures. Our method also innovates the blurring diffusion process with the ResShift mechanism, finely balancing between sharpness and diffusion effects. Comprehensive experiments confirm our framework's advantage over state-of-the-art diffusion-based VSR techniques. The code is available: https://github.com/Tang1705/SeeClear-NeurIPS24.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。