arXiv:2505.16854cs.AIcs.CV2025-05NeurIPS被引 27

让视觉语言模型学会该思考时才思考,大幅减少无用推理

Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models

  • 通过随机屏蔽推理过程,训练模型判断何时需要思考
  • 相比传统方法减少90%的推理长度,性能不降反升
  • 适合追求高效推理的AI研究者和工程师

强化学习(RL)已被证明是提升视觉语言模型(VLMs)推理能力的有效后训练策略。最近的群组相对策略优化(GRPO)方法要求模型在回答前生成完整的推理链,导致词元使用量和计算成本显著增加。受人类思维模式启发——对简单问题直接作答,复杂问题则深入思考——我们探索如何让VLMs先判断是否需要推理。为此,提出TON:一种两阶段训练策略。(i)监督微调阶段采用简单的‘思考丢弃’操作,将推理过程随机替换为空思考,引入‘该思考则思’的初始格式;(ii)GRPO阶段使模型自由探索何时思考或不思考,同时最大化任务相关的奖励。实验表明,与原始GRPO相比,TON可将完成长度降低高达90%,且性能不降甚至提升。在多类任务(包括LLM的GSM8K、VLM的CLEVR/Super-CLEVR/GeoQA、Agentic的AITZ)中,3B和7B模型均显示模型随训练推进逐步跳过不必要的推理步骤。这些发现为实现类人推理路径提供了新方向。代码已开源。

原文摘要 · Abstract (English)

Reinforcement Learning (RL) has proven to be an effective post-training strategy for enhancing reasoning in vision-language models (VLMs). Group Relative Policy Optimization (GRPO) is a recent prominent method that encourages models to generate complete reasoning traces before answering, leading to increased token usage and computational cost. Inspired by the human-like thinking process-where people skip reasoning for easy questions but think carefully when needed-we explore how to enable VLMs to first decide when reasoning is necessary. To realize this, we propose TON, a two-stage training strategy: (i) a supervised fine-tuning (SFT) stage with a simple yet effective 'thought dropout' operation, where reasoning traces are randomly replaced with empty thoughts. This introduces a think-or-not format that serves as a cold start for selective reasoning; (ii) a GRPO stage that enables the model to freely explore when to think or not, while maximizing task-aware outcome rewards. Experimental results show that TON can reduce the completion length by up to 90% compared to vanilla GRPO, without sacrificing performance or even improving it. Further evaluations across LLM (GSM8K), VLM (CLEVR, Super-CLEVR, GeoQA), and Agentic (AITZ) tasks-covering a range of reasoning difficulties under both 3B and 7B models-consistently reveal that the model progressively learns to bypass unnecessary reasoning steps as training advances. These findings shed light on the path toward human-like reasoning patterns in RL approaches. Our code is available at https://github.com/kokolerk/TON.

视觉语言模型强化学习推理优化思维选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。