arXiv:2508.08275cs.CLcs.AI2025-08被引 2

为多模态大模型持续指令微调设计评测基准,可诊断遗忘并评估推理过程质量。

MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis

  • 构建多维度评估框架,结合答案准确率与推理链质量诊断遗忘。
  • 在7个任务6个领域上测试8种算法,发现强基线模型更抗遗忘。
  • 首次将强化微调引入持续学习,带KL控制的GRPO更稳定保留知识。

在后训练阶段进行持续指令微调(CIT)对适应多模态大语言模型(MLLMs)不断变化的实际需求至关重要。然而,由于缺乏严谨且协议一致的评估基准,该领域进展受阻。为此,我们提出MLLM-CTBench,一个涵盖六个不同领域七个挑战性任务的综合性基准。其三大贡献:第一,建立多维度评估框架,联合评估最终答案准确率与过程级推理质量,以思维链(CoT)痕迹作为可观测信号,诊断仅靠答案无法察觉的灾难性遗忘;第二,在统一协议下系统评估来自四大主流家族的八种代表性持续学习算法在不同任务顺序下的表现,为算法设计提供可操作洞察;第三,将范围从监督微调(SFT)扩展至强化微调(RFT),通过研究一种基于策略的强化学习算法GRPO(通过显式KL散度控制稳定更新),分析该机制如何影响跨任务知识保留。实验结果表明:(1) 推理过程质量通常比最终答案准确率更抗遗忘,遗忘主要由领域知识退化驱动;(2) 模型能力是决定持续学习效果的关键因素,更强的基线模型表现出更强的抗遗忘能力;(3) 带有内在KL控制的在策略强化微调(GRPO)相比SFT实现更稳定的跨任务保留,而移除KL控制虽可能提升新任务表现,却会加剧遗忘。

原文摘要 · Abstract (English)

Continual instruction tuning(CIT) during the post-training phase is crucial for adapting multimodal large language models (MLLMs) to evolving real-world demands. However, the progress is hampered by the lack of benchmarks with rigorous, protocol-consistent evaluation. To bridge this gap, we introduce MLLM-CTBench, a comprehensive benchmark for CIT of MLLMs, covering seven challenging tasks across six diverse domains. MLLM-CTBench makes three key contributions. First, we establish a multidimensional evaluation framework that jointly assesses final-answer accuracy and process-level reasoning quality, where Chain-of-Thought (CoT) traces serve as an observable signal to diagnose catastrophic forgetting beyond answer-only evaluation. Second, we conduct a large-scale evaluation of continual learning methods by systematically assessing eight representative algorithms from four major families under a unified protocol across task orders, providing actionable insights for algorithm design. Third, we expand the scope from Supervised Fine-Tuning (SFT) to Reinforcement Fine-Tuning (RFT) in CIT. By investigating GRPO, an on-policy RL algorithm that stabilizes updates through explicit KL-divergence control to a prior policy, we aim to analyze how this mechanism affects cross-task knowledge retention. Our experiments yield several findings:(1) Process-level reasoning quality is often more resilient to catastrophic forgetting than final-answer accuracy, and forgetting is primarily driven by degradation in domain knowledge. (2) Model capability is critical factor influencing continual learning outcomes, with stronger baseline models exhibiting greater resistance to catastrophic forgetting. (3) On-policy RFT (GRPO), with its inherent KL control, achieves more stable cross-task retention than SFT. While removing KL control can amplify forgetting despite potential gains on new ones.

持续学习多模态推理诊断强化微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。