用轻量适配器让冻结的视频生成模型高效实现超分辨率重建。
LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution

- 通过流匹配机制将超分任务转为固定注入模式,降低训练复杂度。
- 仅11.25%可训练参数,单卡A100训练12小时,支持单步采样。
- 适合需要快速部署、低算力要求的视频超分场景。
将大规模预训练视频生成模型适配至新领域的视频超分辨率(VSR)仍面临计算成本高昂的问题。现有方法将生成过程重定义为从低质量到高质量的直接映射,偏离原始生成范式,需大量微调;而类似ControlNet的适配器在现代扩散变压器中效率下降,因缺乏编码器-解码器结构导致骨干网络重复。我们发现流匹配可作为跨域VSR适配的合理替代方案:通过预测所有时间步的恒定速度场,适配任务简化为学习固定注入模式而非时变变换。基于此,我们提出LiteVSR,一个完全冻结扩散变压器的极简框架,搭配轻量级状态感知适配器。该适配器采用双流结构,分别提取低质量输入的静态结构特征与去噪中间状态的动态特征,通过时变交叉注意力对齐,实现去噪过程中从结构对齐到纹理细化的自适应过渡。LiteVSR在仅11.25%可训练参数、单张A100卡12 GPU小时训练下达到竞争力恢复质量,并兼容单步采样。
原文摘要 · Abstract (English)
Adapting large-scale pre-trained video generators for Video Super-Resolution (VSR) in novel domains remains computationally prohibitive. Methods that reformulate generation as direct Low-Quality to High-Quality mappings deviate from the original generative formulation, demanding extensive fine-tuning. ControlNet-style adapters lose their efficiency under modern Diffusion Transformers since the absence of encoder-decoder hierarchy forces duplication of the entire backbone. We observe that flow matching offers a principled alternative for cross-domain VSR adaptation. By predicting a constant velocity field across all timesteps, the adaptation task reduces to learning a fixed injection pattern rather than time-varying transformations. Building on this insight, we propose LiteVSR, a minimalist framework that performs VSR using a completely frozen Diffusion Transformer with a lightweight State-Aware Adapter. The adapter employs a dual-stream architecture that extracts static structural cues from the LQ input and dynamic cues from intermediate denoising states, aligning them through time-dependent cross-attention to enable adaptive transition from structural alignment to texture refinement as denoising proceeds. LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。