arXiv:2605.18852cs.LGcs.AI2026-05被引 4

通过智能评估与稳定性排序,解决多模态大模型训练后期选点难问题。

Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking

论文配图:Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking
图 1 · 摘自论文原文
  • 用多轮采样和分位数聚合提升评估稳定性,避免噪声干扰决策
  • 实测显示选点错误率从32.5%降至11.2%,跨轮次一致性从0.61升至0.84
  • 适合关注模型训练后期优化的开发者和研究者参考

在多模态大语言模型(MLLMs)训练后期,当候选检查点差异微小且下游评估信号存在噪声时,选择最终检查点极具挑战。观测到的微小差异可能与有限评估样本、大模型评判者及模糊多模态证据带来的变异性相当,而验证损失未必反映下游偏好。本文将晚期检查点选择建模为评估不确定性下的稳定性感知决策问题,提出一种结合点级过滤、列表级排序与成对精炼的渐进式框架。通过重复评估集抽样刻画排名稳定性,并采用分位数聚合处理尾部行为。实验表明,多模态数据可评估性至关重要:对含大量OCR输入进行质量感知清洗后,排名翻转率由32.5%降至11.2%,跨运行一致性从0.61提升至0.84。在两个独立的MLLM设置中均观察到验证损失变化与下游偏好不一致。另在11个检查点的Qwen2.5-VL-7B公开复现中,点级得分高度集中,成对比较常出现平局,但多次评估多数情况下选择了中间而非最终检查点。结果表明,可靠的MLLM检查点选择应量化并保留评估不确定性,而非强行依据单次指标微小差异做决策。

原文摘要 · Abstract (English)

Selecting a final checkpoint for multimodal large language models (MLLMs) is challenging when late-stage candidates are closely matched and downstream evaluation signals are noisy. Small observed differences can be comparable to variability introduced by finite evaluation samples, LLM-based judges, and ambiguous multimodal evidence, while validation loss may not identify the checkpoint preferred by downstream evaluation. We formulate late-stage checkpoint selection as a stability-aware decision problem under evaluation uncertainty and propose a progressive framework combining pointwise filtering, listwise ranking, and pairwise refinement. Repeated evaluation-set subsampling is used to characterize ranking stability, while percentile-based aggregation accounts for lower- and upper-tail behavior. Experiments show that multimodal data evaluability is critical: quality-aware curation of OCR-heavy inputs reduces ranking flip rate from 32.5\% to 11.2\% and increases inter-run agreement from 0.61 to 0.84. We further observe divergence between validation-loss progression and downstream checkpoint preference in two independent MLLM settings. An additional public Qwen2.5-VL-7B reproduction across 11 checkpoints shows tightly clustered pointwise scores and frequently tie-dominated final pairwise comparisons, while repeated evaluation most often selects an intermediate rather than the final checkpoint. These results suggest that reliable MLLM checkpoint selection should quantify and reserve evaluation uncertainty rather than force decisions from small differences in a single metric.

多模态模型训练优化评估稳定性检查点选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。