动态路由让多模态视频生成更灵活,支持文本、图像、视频混合输入。
MoRoute: Dynamic Routing for In-Context Multimodal Video Generation

- 通过轻量级路由选择最相关的视觉语言层,适配不同生成阶段。
- 在多个基准上平均得分提升0.15至0.34,优于现有方法。
- 支持图像和视频直接作为上下文条件,保留细节信息,适合编辑任务。
多模态视频生成旨在单一模型中基于任意组合的文本、图像和视频条件生成与编辑视频,使多样化任务共享互补数据与生成先验。统一此类任务需要对多样条件进行多模态理解,通常由预训练视觉-语言模型(VLM)提供。核心挑战在于如何将异构的VLM分层表示与预训练视频扩散变换器(DiT)连接。现有方法或仅注入最终层或少数手动选择的VLM层,或联合训练架构匹配的理解与生成流,难以复用异构预训练主干。我们提出MoRoute,一个统一的多模态视频生成框架,将冻结的VLM与不同架构的预训练视频DiT视为异构专家,通过动态层路由连接。对于每个输入,轻量级块级路由器使每个DiT块可选择最相关于其生成阶段的VLM层,从而学习自适应的多模态理解与视频合成对应关系。MoRoute进一步通过统一的上下文条件,将参考图像和源视频直接嵌入DiT标记序列,保留跨多样化生成与编辑任务的细粒度视觉细节。在IntelligentVBench、OpenVE-Bench和RefVIE-Bench上的实验表明,MoRoute在每个基准上均持续超越最佳对比方法,平均得分分别提升0.15、0.18和0.34(满分5分)。
原文摘要 · Abstract (English)
Multimodal video generation aims to generate and edit videos conditioned on arbitrary combinations of text, images, and videos within a single model, allowing diverse tasks to share complementary data and generative priors. Unifying these tasks requires multimodal understanding of diverse conditions, which is typically provided by a pretrained vision-language model (VLM). A key challenge is how to connect the VLM's hierarchical multimodal representations with a pretrained video diffusion transformer (DiT). Existing methods either inject features from only the final or a few manually selected VLM layers, or jointly train architecture-matched understanding and generation streams, making it difficult to reuse heterogeneous pretrained backbones. We introduce MoRoute, a unified multimodal video generation framework that formulates a frozen VLM and a pretrained video DiT with different architectures as heterogeneous experts connected through dynamic layer routing. For each input, a lightweight block-wise router enables every DiT block to select the VLM layer most relevant to its generation stage, thereby learning an adaptive correspondence between multimodal understanding and video synthesis. MoRoute further incorporates reference images and source videos directly into the DiT token sequence through unified in-context conditioning, preserving fine-grained visual details across diverse generation and editing tasks. Experiments on IntelligentVBench, OpenVE-Bench, and RefVIE-Bench show that MoRoute consistently surpasses the best competing method on each benchmark, improving the average score by 0.15, 0.18, and 0.34 on a 1-5 scale, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。