提出MoDE架构，解决多模态模型持续学习中的模态内与跨模态遗忘问题。
Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
- 通过解耦模态更新，减少不同模态间的梯度冲突
- 在多个基准上显著降低跨模态和模态内遗忘率
- 轻量级设计适合大规模多模态模型持续学习
统一多模态生成模型（UMGMs）将视觉理解与图像生成整合于单一自回归框架中。然而，其持续学习新任务的能力严重受限于灾难性遗忘，包括模态内（intra-modal）和跨模态（inter-modal）遗忘。尽管模态内遗忘已有研究，跨模态遗忘仍基本未被探索。本文首次在UMGMs中识别并实证验证该现象，并从模态间梯度冲突的理论角度提供解释。为此，我们提出轻量且可扩展的模态解耦专家（Modality-Decoupled Experts, MoDE）架构：通过隔离各模态专属更新以缓解梯度冲突，并利用知识蒸馏防止灾难性遗忘、保留预训练能力。相比以往保持模态耦合的方法，MoDE显式解耦模态以避免干扰。在多样基准上的实验表明，MoDE显著缓解了跨模态与模态内遗忘，在统一多模态生成场景中优于现有持续学习基线。代码将公开：https://github.com/Christina200/MoDE-official.git
原文摘要 · Abstract (English)
Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。