arXiv:2603.23326cs.CV2026-03被引 2

用纯图像训练出超高清视频生成模型,无需视频数据。

ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images

  • 分两阶段适配:先用低分辨率图桥接图像视频差异,再用高分辨率图扩展空间细节。
  • 在VBench上达0.8分提升,超清视频细节丰富,无需视频训练数据。
  • 适合想用图像模型生成高质量视频的研究者或开发者。

基于Transformer的视频扩散模型依赖时空令牌的3D注意力,导致时间与内存复杂度呈二次增长,使超高清视频的端到端训练代价高昂。为突破此瓶颈,我们提出纯图像适配框架,将原生尺度预训练的视频扩散Transformer升级为生成更高分辨率视频的能力。然而,仅用高分辨率图像微调常因图像-视频模态差距引入明显噪声。为此,我们解耦学习目标,分别处理模态对齐与空间外推。核心是两阶段适配策略——继洛拉(Relay LoRA):第一阶段用低分辨率图像将视频模型适配至图像域以弥合模态差异;第二阶段用高分辨率图像进一步适配以获得空间外推能力。推理时仅保留高分辨率适配部分,保持视频生成模态同时实现超清视频合成。为增强细粒度细节生成,我们提出高频感知训练目标,通过专用重建损失显式引导模型从退化潜在表示中恢复高频成分。大量实验表明,本方法无需任何视频训练数据即可生成具有丰富视觉细节的超高清视频,甚至在VBench基准上超越此前基于高分辨率视频训练的SOTA模型0.8分。

原文摘要 · Abstract (English)

Transformer-based video diffusion models rely on 3D attention over spatial and temporal tokens, which incurs quadratic time and memory complexity and makes end-to-end training for ultra-high-resolution videos prohibitively expensive. To overcome this bottleneck, we propose a pure image adaptation framework that upgrades a video Diffusion Transformer pre-trained at its native scale to synthesize higher-resolution videos. Unfortunately, naively fine-tuning with high-resolution images alone often introduces noticeable noise due to the image-video modality gap. To address this, we decouple the learning objective to separately handle modality alignment and spatial extrapolation. At the core of our approach is Relay LoRA, a two-stage adaptation strategy. In the first stage, the video diffusion model is adapted to the image domain using low-resolution images to bridge the modality gap. In the second stage, the model is further adapted with high-resolution images to acquire spatial extrapolation capability. During inference, only the high-resolution adaptation is retained to preserve the video generation modality while enabling high-resolution video synthesis. To enhance fine-grained detail synthesis, we further propose a High-Frequency-Awareness-Training-Objective, which explicitly encourages the model to recover high-frequency components from degraded latent representations via a dedicated reconstruction loss. Extensive experiments demonstrate that our method produces ultra-high-resolution videos with rich visual details without requiring any video training data, even outperforming previous state-of-the-art models trained on high-resolution videos by 0.8 on the VBench benchmark. Code will be available at https://github.com/WillWu111/ViBe.

视频生成扩散模型图像适配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。