让多模态视频生成利用不同模型的专长，提升画质与连贯性。
Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
- 分离扩散模型中的模态特征，分别对齐到对应专家模型。
- 在多个数据集上显著提升视频视觉质量与长期一致性。
- 适合做多模态视频生成、融合多种预训练模型的研究者。
新兴的多模态世界模型尝试联合生成多种模态（如RGB、深度图、掩码）的视频，但未能充分利用现有基础模型的丰富先验知识。我们提出M²-REPA，首个专为多模态视频生成设计的表示对齐方法。核心思想是：在不同模态空间训练的基础模型天然捕捉到特定领域的先验，可视为互补的“专家”。具体而言，我们首先从扩散模型的中间表示中解耦出模态特异性特征，再将其分别与对应专家模型对齐。为此，我们设计了两个协同优化的目标：多模态表示对齐损失，强制特征与专家匹配；模态特异性解耦正则化，鼓励不同模态间的互补性。该设计实现联合优化，充分挖掘多个基础模型的先验优势。大量实验证明，本方法在视觉质量和长期一致性方面显著优于基线。
原文摘要 · Abstract (English)
Emerging multi-modal world models attempt to jointly generate videos across diverse modalities (e.g., RGB, depth, and mask), yet they fail to fully exploit the rich priors of existing foundation models. We propose $M^2$-REPA, the first representation alignment method tailored for multi-modal video generation. Our key insight is that foundation models trained on different modality spaces naturally capture distinct domain-specific priors, acting as complementary "experts." Specifically, we first decouple modality-specific features from the diffusion model's intermediate representations, then align each with its corresponding expert foundation model. To this end, we design two synergistic objectives: a multi-modal representation alignment loss that enforces feature-to-expert matching, and a modality-specific decoupling regularization that encourages complementarity across different modalities. This design enables joint optimization, fully exploiting priors from multiple foundation models. Extensive experiments demonstrate that our method significantly outperforms baselines in visual quality and long-term consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。