arXiv:2510.01581cs.LGcs.AI2025-10被引 3

让模型学会根据题目难易自动调节思考长度,避免想得不够或过度纠结。

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

  • 通过自注意力识别关键推理步骤,动态剪枝冗余内容。
  • 在多项任务上提升8.4%准确率,推理步数减少36.8%。
  • 适合需要高效精准推理的应用,如数学竞赛与复杂问答。

近期的思维模型通过增加测试时计算量来解决复杂推理任务,但计算资源需根据题目难度合理分配。过短的推理(思考不足)会导致难题出错;而过长的推理(过度思考)则会生成冗余步骤,浪费token。本文称此为适应性不足问题。为此提出TRAAC(Think Right with Adaptive, Attentive Compression),一种在线后训练强化学习方法,利用模型对长推理轨迹的自注意力机制识别重要步骤并剪枝冗余部分。TRAAC还估计题目难度,并将其纳入奖励函数,使模型学会按难度分配推理预算。在AIME、AMC、GPQA-D、BBEH等任务上,TRAAC(Qwen3-4B)相比基线模型平均准确率提升8.4%,推理步数减少36.8%;相比最优强化学习基线,准确率提升7.9%,步数减少29.4%。该方法还展现出强泛化能力:尽管训练仅在数学数据集上进行,但在非数学数据集如GPQA-D、BBEH和OptimalThinkingBench上也取得准确率与效率双提升。分析表明,TRAAC能根据难度精细调节思考预算,任务难度校准与注意力压缩结合可带来跨任务增益。

原文摘要 · Abstract (English)

Recent thinking models solve complex reasoning tasks by scaling test-time compute, but this scaling must be allocated in line with task difficulty. On one hand, short reasoning (underthinking) leads to errors on harder problems that require extended reasoning steps; but, excessively long reasoning (overthinking) can be token-inefficient, generating unnecessary steps even after reaching a correct intermediate solution. We refer to this as under-adaptivity, where the model fails to modulate its response length appropriately given problems of varying difficulty. To address under-adaptivity and strike a balance between under- and overthinking, we propose TRAAC (Think Right with Adaptive, Attentive Compression), an online post-training RL method that leverages the model's self-attention over a long reasoning trajectory to identify important steps and prune redundant ones. TRAAC also estimates difficulty and incorporates it into training rewards, thereby learning to allocate reasoning budget commensurate with example difficulty. Our approach improves accuracy, reduces reasoning steps, and enables adaptive thinking compared to base models and other RL baselines. Across a variety of tasks (AIME, AMC, GPQA-D, BBEH), TRAAC (Qwen3-4B) achieves an average absolute accuracy gain of 8.4% with a relative reduction in reasoning length of 36.8% compared to the base model, and a 7.9% accuracy gain paired with a 29.4% length drop compared to the best RL baseline. TRAAC also shows strong generalization: although our models are trained on math datasets, they show accuracy and efficiency gains on out-of-distribution non-math datasets like GPQA-D, BBEH, and OptimalThinkingBench. Our analysis further verifies that TRAAC provides fine-grained adjustments to thinking budget based on difficulty and that a combination of task-difficulty calibration and attention-based compression yields gains across diverse tasks.

推理优化强化学习自注意力思维压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。