提出可组合的多模态预训练框架，解决新增模态时遗忘旧知识的问题。
Rosetta: Composable Native Multimodal Pretraining
- 核心知识保留在共享专家中，新模态用即插即用专家扩展。
- 在新增图像生成任务后仍保持原有语言与视觉理解能力。
- 适合需要持续扩展模态的研究者，尤其关注模型稳定性与可扩展性。
实现真正的人工通用智能需要能够无缝整合新模态且不遗忘已有知识的基础模型。然而，将连续生成目标与离散理解任务并行处理会导致严重的梯度冲突。现有架构（如标准 Mixture-of-Experts, MoE）极易发生表征覆盖。即使结构上分区的方案（如 Mixture-of-Transformers, MoT）仍面临灾难性遗忘，严重制约多模态扩展。本文提出 Rosetta，一种可组合的原生多模态预训练框架，支持无损模态扩展。Rosetta 采用模块化设计：核心基础知识由全局共享专家保留，模态特异性能力分布于即插即用专家中。为确保无损组合，提出动量锚定正交投影（MAOP），利用优化器动量状态作为隐式语义锚点，选择性中和新模态带来的冲突梯度，同时保留协同更新。大量实验表明，标准 MoE 与 MoT 在新增模态时严重遗忘已有知识，而 Rosetta 能稳健保持既有的语言与视觉理解能力，并显著提升图像生成性能，激发跨模态协同效应，为真正可组合、统一的多模态基础模型铺平道路。项目代码与检查点已开源，详见 https://rosetta-lmm.github.io/。
原文摘要 · Abstract (English)
Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. However, accommodating continuous generative objectives alongside discrete understanding tasks causes severe gradient conflicts. Existing architectures, including standard Mixture-of-Experts (MoE), are highly susceptible to representation overwriting. Even structurally partitioned paradigms like Mixture-of-Transformers (MoT) remain vulnerable to catastrophic forgetting, severely impeding multimodal scalability. In this work, we introduce Rosetta, a composable native multimodal pretraining framework designed for seamless and non-destructive modality expansion. Rosetta adopts a modular paradigm where core foundational knowledge is preserved within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts. To guarantee non-destructive composition, we propose Momentum-Anchored Orthogonal Projection (MAOP). MAOP leverages the optimizer's momentum state as an implicit semantic anchor, selectively neutralizing conflicting gradient components from new modalities while preserving synergistic updates. Extensive evaluations demonstrate that, while standard MoE and MoT architectures suffer catastrophic forgetting of previously acquired knowledge, Rosetta robustly preserves established language and visual understanding. Furthermore, it delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models. To facilitate further multimodal research, we release our code and checkpoints to the community. Project page at https://rosetta-lmm.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。