提出分层解耦方法，让多模态大模型持续学习新指令时兼顾效率与性能。
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
- 按层分析模型对不同任务的响应差异，动态分离任务特异性与通用性特征。
- 在新基准上测试，相比现有方法准确率提升超过5%且内存占用更低。
- 适合需要长期更新指令能力的多模态系统开发者使用。
指令微调广泛用于通过特定任务数据集训练预训练多模态大语言模型（MLLM），以增强对人类指令的理解。然而，在真实场景中难以一次性收集所有可能的指令数据集，因此使MLLM具备持续指令微调能力至关重要。但现有方法常在内存效率与性能间权衡，严重影响整体效率。本文提出一种基于中心核对齐（CKA）相似性在不同模型层上的变化，实现任务特异性扩展与任务通用性融合的框架。此外，我们分析了现有基准中的信息泄露问题，提出一个更严苛的新基准以合理评估各方法性能。大量实验表明，本方法显著优于现有最先进方法。代码与数据集已开源：https://github.com/Ghy0501/HiDe-LLaVA。
原文摘要 · Abstract (English)
Instruction tuning is widely used to improve a pre-trained Multimodal Large Language Model (MLLM) by training it on curated task-specific datasets, enabling better comprehension of human instructions. However, it is infeasible to collect all possible instruction datasets simultaneously in real-world scenarios. Thus, enabling MLLM with continual instruction tuning is essential for maintaining their adaptability. However, existing methods often trade off memory efficiency for performance gains, significantly compromising overall efficiency. In this paper, we propose a task-specific expansion and task-general fusion framework based on the variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. Furthermore, we analyze the information leakage present in the existing benchmark and propose a new and more challenging benchmark to rationally evaluate the performance of different methods. Comprehensive experiments showcase a significant performance improvement of our method compared to existing state-of-the-art methods. Code and dataset are released at https://github.com/Ghy0501/HiDe-LLaVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。