用可学习的路由机制，让多模态扩散模型更精准地融合图文信息。
Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
- 通过动态路由选择关键隐藏状态，实现跨模态特征对齐。
- 仅用30亿~50亿参数，性能媲美甚至超越四倍大的模型。
- 适合追求高效推理与高精度生成的多模态应用开发。
我们提出MoS（Mixture of States），一种用于多模态扩散模型的新颖融合范式，通过灵活的状态级交互合并不同模态。其核心是一个可学习的、逐标记的路由器，能根据去噪时间步和输入内容，动态建立模态间隐藏状态的依赖关系，精确对齐标记级特征与扩散轨迹。该路由器稀疏选择前k个隐藏状态，并采用ε-贪婪策略训练，以极少可学习参数和几乎可忽略的计算开销，高效选出上下文相关特征。我们在文本到图像生成（MoS-Image）和编辑（MoS-Editing）任务上验证了设计有效性，模型在仅30亿至50亿参数下达到或超过最大4倍规模模型的性能。这些结果确立了MoS作为可扩展且计算高效的多模态扩散模型融合范式。
原文摘要 · Abstract (English)
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-$k$ hidden states and is trained with an $ε$-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to $4\times$ larger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。