arXiv:2506.08125cs.LGcs.CL2025-06ACL被引 6

通过识别无关紧要的词元,让大模型更高效地推理。

Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning

  • 根据词元重要性设计奖励机制,精准惩罚冗余内容。
  • 响应长度显著减少,正确率反而提升或保持不变。
  • 适合追求高效推理的大模型应用开发者。

大语言模型虽具备强大推理能力,但常生成冗长解释,降低效率。尽管强化学习已被用于优化推理,但多数方法仅关注准确率,依赖统一长度奖励,忽略单个词元的贡献差异,反而损害正确性。本文从词元重要性角度重审强化学习中的长度优化问题。观察到许多思维链(CoT)词元对最终答案贡献甚微,提出一种显著性感知的长度奖励,选择性惩罚不重要的词元,减少冗余同时保留关键推理。还设计动态长度奖励,在训练初期鼓励详细推理,后期逐步转向简洁。将这些组件融入标准策略优化框架,实现推理效率与准确率的双重提升。在多个基准测试中,响应长度大幅减少,正确率维持或提高,凸显建模词元重要性的关键作用。

原文摘要 · Abstract (English)

Large language models (LLMs) show strong reasoning abilities but often produce unnecessarily long explanations that reduce efficiency. Although reinforcement learning (RL) has been used to improve reasoning, most methods focus on accuracy and rely on uniform length-based rewards that overlook the differing contributions of individual tokens, often harming correctness. We revisit length optimization in RL through the perspective of token significance. Observing that many chain-of-thought (CoT) tokens contribute little to the final answer, we introduce a significance-aware length reward that selectively penalizes insignificance tokens, reducing redundancy while preserving essential reasoning. We also propose a dynamic length reward that encourages more detailed reasoning early in training and gradually shifts toward conciseness as learning progresses. Integrating these components into standard policy optimization yields a framework that improves both reasoning efficiency and accuracy. Experiments across multiple benchmarks demonstrate substantial reductions in response length while preserving or improving correctness, highlighting the importance of modeling token significance for efficient LLM reasoning.

大模型推理强化学习效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。