对比早融合与晚融合架构，发现早融合更高效且性能更强。
Scaling Laws for Native Multimodal Models
- 采用早融合架构，直接联合训练多模态信号，不依赖预训练编码器。
- 在低参数量下早融合表现更优，训练与部署效率更高。
- 引入专家混合机制，提升模型对不同模态的适应能力。
构建能通过多模态信号有效感知世界的通用模型是长期目标。现有方法通常将独立预训练组件（如视觉编码器与大语言模型）连接后继续多模态训练，虽具优异样本效率，但其是否天然优于其他架构仍不明。本文重新审视从零开始训练的原生多模态模型（NMMs），开展大规模缩放定律研究，涵盖457个不同架构与训练组合的模型。结果表明，早融合架构并无本质劣势，反而在参数量较低时表现更佳，训练更高效，部署更简便。基于此，我们引入专家混合（MoE）机制，使模型能学习模态特定权重，显著提升性能。
原文摘要 · Abstract (English)
Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately pre-trained components, such as connecting vision encoders to LLMs and continuing multimodal training. While such approaches exhibit remarkable sample efficiency, it remains an open question whether such late-fusion architectures are inherently superior. In this work, we revisit the architectural design of native multimodal models (NMMs)-those trained from the ground up on all modalities-and conduct an extensive scaling laws study, spanning 457 trained models with different architectures and training mixtures. Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones, which do not rely on image encoders or tokenizers. On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy. Motivated by the strong performance of the early-fusion architectures, we show that incorporating Mixture of Experts (MoEs) allows models to learn modality-specific weights, significantly benefiting performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。