用视频生成模型统一视频理解与生成,提升效率与性能。
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

- 以视频生成模型为基底,构建统一框架。
- 双向训练使生成知识反哺理解,性能媲美专用模型。
- 适合需要多模态统一能力的研究者和开发者。
统一多模态模型在视觉理解与生成融合中面临根本挑战:视频生成的计算成本远高于理解。为此,我们提出逆向传统范式的方法:不将理解主导的多模态大模型扩展为生成模型,而是以视频生成器为基础构建统一框架——Uni-ViGU。引入统一流方法,在同一过程中对视频进行连续流匹配,对文本进行离散流匹配,实现多模态生成的一致性。进一步设计一种模态驱动的MoE架构,在Transformer块中加入轻量级文本生成层,同时保留生成先验。为将生成知识用于理解,提出双向训练机制:第一阶段知识回溯通过重建输入提示来利用已学文本-视频对应关系;第二阶段能力精炼在详细描述上微调,建立判别性共享表示。实验表明,Uni-ViGU在视频生成与理解任务上均达到竞争力表现,验证了以生成为中心的架构是迈向统一多模态智能的可扩展路径。
原文摘要 · Abstract (English)
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。