arXiv:2507.08806cs.AIcs.CL2025-07EMNLP被引 20

通过删减冗余思维片段,让大模型更清晰地推理。

Think Clearly: Improving Reasoning via Redundant Token Pruning

  • 用特殊结束标记衡量每步思考的注意力,定位冗余内容。
  • 结构化剪枝优先删除贡献低的推理段,提升准确性。
  • 无需训练,在数学竞赛题上表现优异,适合逻辑推理任务。

近期大型语言模型在长序列推理中展现出潜力,遵循结构化的思维链得出最终答案。然而我们发现这些推理路径存在大量冗余;分析注意力模式显示注意力得分分布广泛,错误答案尤其表现出更强的稀疏性。本文表明,有意识地移除推理过程中的冗余能显著提升性能,即通过清除干扰实现清晰思考。具体而言,我们通过测量每个标记对特殊结束思考标记的注意力得分,系统性识别推理冗余,并插入显式指令以标记每一步推理的结束。此外,提出结构感知剪枝策略,优先剔除低贡献的推理块而非单个标记。剔除冗余后,移除注入的结束指令,继续生成推理。该方法在无需训练的情况下显著提升多个推理密集型基准的准确率,尤其在数学竞赛数据集AIME和AMC上表现突出,其中推理冗余更为普遍。

原文摘要 · Abstract (English)

Recent large language models have shown promising capabilities in long-form reasoning, following structured chains of thought before arriving at a final answer. However, we observe that these reasoning paths tend to include substantial redundancy; analyzing attention patterns reveals that attention scores are widely scattered, particularly incorrect answers exhibit greater attention sparsity. In this paper, we demonstrate that deliberately removing this redundancy in the reasoning process significantly improves performance through clear thinking, i.e., removing distraction. Specifically, we systematically identify reasoning redundancy by measuring token-level attention scores to a special end-of-thinking token, which is appended to an explicit instruction inserted to conclude each intermediate reasoning step. Furthermore, we propose structure-aware pruning that prioritizes removing tokens in low-contributing reasoning chunks over individual tokens. After evicting redundant tokens, we remove the injected end-of-thinking instruction, then resume the reasoning generation. We demonstrate that our method significantly improves overall accuracy across reasoning-intensive benchmarks without any training involved. In particular, our method shows strong performance on challenging mathematical competition benchmarks such as AIME and AMC, where reasoning redundancy is more prevalent.

推理优化注意力剪枝数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。