让视觉语言模型持续学习新任务，不丢旧能力。
LLaVA-c: Continual Improved Visual Instruction Tuning
- 用谱感知加固和无监督探问正则化，解决任务失衡与过拟合问题。
- 持续学习下性能超越多任务联合训练，通用能力不下降。
- 适合需要长期迭代更新的多模态应用开发人员。
如LLaVA-1.5的多模态模型通过在多任务数据集上进行视觉指令微调，实现了顶尖的视觉理解能力，具备出色的指令遵循与多模态表现。然而，多任务学习面临任务平衡难题，需精细调节数据比例，且扩展成本高——新增任务易引发灾难性遗忘，需昂贵重训。持续学习提供了一种增量获取知识的替代方案，但现有方法侧重特定任务性能，忽视基础模型因过度拟合特定指令而产生的泛化能力退化。本文在LLaVA-1.5基础上提出两项简单但有效改进：谱感知加固以改善任务平衡，无监督探问正则化防止基础模型退化。我们在持续预训练与微调场景下评估了通用与任务特定性能。实验表明，LLaVA-c持续提升标准基准表现并保留通用能力。首次证明，逐任务持续学习可达到甚至超越多任务联合学习效果。代码将公开发布。
原文摘要 · Abstract (English)
Multimodal models like LLaVA-1.5 achieve state-of-the-art visual understanding through visual instruction tuning on multitask datasets, enabling strong instruction-following and multimodal performance. However, multitask learning faces challenges such as task balancing, requiring careful adjustment of data proportions, and expansion costs, where new tasks risk catastrophic forgetting and need costly retraining. Continual learning provides a promising alternative to acquiring new knowledge incrementally while preserving existing capabilities. However, current methods prioritize task-specific performance, neglecting base model degradation from overfitting to specific instructions, which undermines general capabilities. In this work, we propose a simple but effective method with two modifications on LLaVA-1.5: spectral-aware consolidation for improved task balance and unsupervised inquiry regularization to prevent base model degradation. We evaluate both general and task-specific performance across continual pretraining and fine-tuning. Experiments demonstrate that LLaVA-c consistently enhances standard benchmark performance and preserves general capabilities. For the first time, we show that task-by-task continual learning can achieve results that match or surpass multitask joint learning. The code will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。