优化推理模型的思考标记，提升效率不降性能。
Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model
- 设计双策略偏好优化算法，平衡有无思考标记的响应
- 在5个数学推理基准上提升令牌效率，保持模型性能
- 适合关注大模型推理效率的开发者和研究者
大型推理模型（LRMs）在解决复杂问题上表现优异，但在处理简单任务时常产生冗长且充满思考标记（如“等等”、“然而”）的回应。这些标记会触发不必要的高阶推理行为（如反思、回溯），降低效率。本研究发现，这类由思考标记引发的行为并非有效解题所必需，反而在有限令牌预算下可能阻碍正确推理，称之为“思考陷阱”。为此，提出双策略偏好优化（DuP-PO）算法：(1) 滚动采样策略，确保有与无思考标记响应的均衡暴露；(2) 细粒度优势控制技术，动态调节目标标记预测；(3) 策略塑形方法，保障思考标记梯度贡献稳定。在五个主流数学推理基准上的实验表明，该方法显著提升了主流大型推理模型的令牌效率，同时保持甚至优于基线模型性能。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) excel at solving complex problems but face an overthinking dilemma. When handling simple tasks, they often produce verbose responses overloaded with thinking tokens (e.g., wait, however). These tokens trigger unnecessary high-level reasoning behaviors like reflection and backtracking, reducing efficiency. In this work, our pilot study reveals that these thinking-token-induced behaviors are not essential for effective problem-solving and may even hinder correct reasoning within constrained token budgets. We identify this phenomenon as the thinking trap. To mitigate this issue, we propose Dual Policy Preference Optimization (DuP-PO), a novel algorithm featuring: (1) A rollout sampling strategy that guarantees balanced exposure to responses with and without thinking tokens; (2) A fine-grained advantage control technique to dynamically regulate the prediction of target tokens; (3) A policy shaping method ensuring stable gradient contributions from thinking tokens. Experimental results on five popular math reasoning benchmarks show that DuP-PO performs well on the popular LRM, which significantly improves their token efficiency during reasoning, while achieving superior performance of the base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。