让视觉语言模型的推理更简洁准确,训练时间减少七成
Balanced Thinking: Improving Chain of Thought Training in Vision Language Models
- 用动态权重区分思考与答案段,逐步聚焦答案
- 在多个基准上准确率超传统微调,仅需1/7训练时长
- 适合追求高效高精度推理的开发者和研究者
视觉语言模型的多模态推理通常采用两阶段流程:监督微调(SFT)和强化学习(RL)。标准SFT中所有标记对损失贡献相同,但推理数据天然存在标记不平衡问题。较长的<think>推理过程会掩盖短但关键的<answer>部分,导致推理冗长且答案不准确。本文提出SCALe(调度式课程自适应损失),通过动态、长度无关的加权机制,明确分离对推理与答案段的监督。与常规SFT过度强调<think>不同,SCALe-SFT利用余弦调度策略,在训练过程中逐步将关注点从<think>转移至<answer>,鼓励简洁且有依据的推理。我们在多种基准和架构上评估了SCALe,结果表明其性能持续优于标准SFT,达到完整两阶段SFT+GRPO流程的水平,同时仅需约1/7的训练时间,是一种轻量高效的替代方案。结合GRPO后,SCALe实现最佳整体表现,凸显其作为独立方法或强化精炼基础的价值。
原文摘要 · Abstract (English)
Multimodal reasoning in vision-language models (VLMs) typically relies on a two-stage process: supervised fine-tuning (SFT) and reinforcement learning (RL). In standard SFT, all tokens contribute equally to the loss, even though reasoning data are inherently token-imbalanced. Long <think> traces overshadow short but task-critical <answer> segments, leading to verbose reasoning and inaccurate answers. We propose SCALe (Scheduled Curriculum Adaptive Loss), which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting. Unlike vanilla SFT, which overweights the <think> segment, SCALe-SFT gradually shifts the focus from <think> to <answer> throughout training via a cosine scheduling policy, encouraging concise and well-grounded reasoning. We evaluate SCALe across diverse benchmarks and architectures. Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline while requiring only about one-seventh of the training time, making it a lightweight yet effective alternative. When combined with GRPO, SCALe achieves the best overall performance, highlighting its value both as a standalone method and as a strong foundation for reinforcement refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。