arXiv:2503.16188cs.CV2025-03被引 45

提出无需思考的强化微调方法,发现小模型不思考反而更有效。

Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuning

  • 用简单准确率奖励替代思维过程,实现无思考强化微调
  • 小模型在无思考下性能优于有思考,跨任务表现稳定
  • 模型可自适应决定是否思考,提升效率与效果

本文研究规则驱动的视觉强化微调(RFT)中显式思维过程的作用。提出CLS-RL用于多模态大模型图像分类,使用可验证奖励进行微调,实验显示其显著优于SFT并具备跨数据集泛化能力。进一步质疑显式思维是否必要,引入无思考强化微调(No-Thinking-RL),通过简单等式准确率奖励实现无思维微调。在6个不同任务、多种模型规模和类型的实验中发现:1)视觉感知任务无需思维,无思考方法在各模型规模上持续优于或匹配有思维方法;2)能力有限的模型难以生成高质量思维链,导致有思维方法效果较差;3)部分有思维方法的回答存在思维内容与答案标签不一致现象,整体准确率低于平均值。推测显式思维可能阻碍奖励收敛。为此提出先答后思(Think-After-Answer)策略进行验证。最后开展试点研究,引入自适应思考方法(Adaptive-Thinking),实验表明该方法能根据模型能力和任务复杂度收敛至特定提示,性能媲美或优于有/无思考方法,表明多模态大模型可自主判断是否需要思考。

原文摘要 · Abstract (English)

This paper investigates the role of explicit thinking process in rule-based reinforcement fine-tuning (RFT) for MLLMs. We first propose CLS-RL for MLLM image classification, using verifiable rewards for fine-tuning. Experiments show CLS-RL significantly outperforms SFT and yields a cross-dataset generalization effect. We then rethink and question whether explicit thinking in RFT is always necessary. Challenging the convention that explicit thinking is crucial for the success of RFT, we introduce No-Thinking-RL, exploring RFT without thinking by introducing a simple equality accuracy reward. We evaluate No-Thinking-RL on 6 diverse tasks across different model sizes and types. Experimental results reveal three key findings: 1). Visual perception tasks do not require thinking during RFT, as No-Thinking-RL consistently outperforms or matches Thinking-based RFT across model sizes. 2).} Models with limited capabilities struggle to generate high-quality CoT for RFT, making Thinking-based RFT less effective than No-Thinking-RL. 3). There are inconsistencies between the answers in the thinking and answer tags for some responses of thinking-based RFT, which show lower accuracy than the overall accuracy. We hypothesize that explicit thinking before verifiable answers may hinder reward convergence and reduce performance. To test this hypothesis, we propose Think-After-Answer, which places thinking after the answer to mitigate this effect for experimental verification. Lastly, we conduct a pilot study to explore whether MLLMs can learn when to think during RFT, introducing an Adaptive-Thinking method. Experiments show that it converges to a specific prompt depending on model capability and task complexity, achieving comparable or better performance than both Thinking and No-Thinking-RL. This suggests MLLMs can adaptively decide to think or not based on their capabilities and task complexity.

强化学习大模型自适应推理微调方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。