简单调整训练策略就能避免多模态大模型遗忘，效果优于复杂方法。
Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
- 通过控制参数量和学习率缓解分布外图像的遗忘问题。
- 发现分布内图像+分布外文本时存在任务特异性过拟合现象。
- 引入数据混合训练策略，自然适配持续学习场景。
本文表明，对多模态大语言模型（MLLM）仅做简单的微调策略调整即可有效缓解灾难性遗忘。在视觉问答任务中，我们设计了一个2×2实验框架，评估模型在分布内与分布外图像及文本输入下的表现。结果表明，适当的正则化（如限制可训练参数数量或采用低学习率）能有效防止分布外图像引发的遗忘。然而，我们发现当图像为分布内而文本为分布外时，会出现一种独特的遗忘现象，归因于任务特异性过拟合。为此，我们提出一种数据混合训练策略，融合不同数据集与任务以解决该问题。最终，该方法可自然扩展至持续学习场景，在性能上超越现有依赖复杂辅助机制的方法。总体而言，研究挑战了当前主流假设，揭示了MLLM固有的鲁棒性，并提供了保留其通用能力的实用指导。
原文摘要 · Abstract (English)
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。