用信息量指导推理，让模型少说废话但更准。
IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning
- 根据每个词元与答案的互信息分配奖励，精准识别有用推理步骤。
- 推理长度最多减少36%，准确率反而提升，且不降低正确性。
- 适合追求高效推理的AI研发者，尤其关注资源优化的应用场景。
大型语言模型依赖长链式思维提升准确率，但带来高昂的推理成本。本文重新审视令牌高效的后训练方法，指出现有序列级奖励调节方法对推理努力分配的控制有限。为此提出IAPO，一种基于信息论的后训练框架，通过计算每个词元与最终答案的条件互信息（MI），为其分配词元级优势。该方法能明确识别有信息量的推理步骤，抑制低效探索。理论分析表明，IAPO可实现推理冗余的单调减少而不损害正确性。实验证明，IAPO在多个推理数据集上均显著提升准确率，并将推理长度最多缩短36%，优于现有令牌高效强化学习方法。大量评估验证了信息感知的优势塑造是令牌高效后训练的有力通用方向。代码已开源。
原文摘要 · Abstract (English)
Large language models increasingly rely on long chains of thought to improve accuracy, yet such gains come with substantial inference-time costs. We revisit token-efficient post-training and argue that existing sequence-level reward-shaping methods offer limited control over how reasoning effort is allocated across tokens. To bridge the gap, we propose IAPO, an information-theoretic post-training framework that assigns token-wise advantages based on each token's conditional mutual information (MI) with the final answer. This yields an explicit, principled mechanism for identifying informative reasoning steps and suppressing low-utility exploration. We provide a theoretical analysis showing that our IAPO can induce monotonic reductions in reasoning verbosity without harming correctness. Empirically, IAPO consistently improves reasoning accuracy while reducing reasoning length by up to 36%, outperforming existing token-efficient RL methods across various reasoning datasets. Extensive empirical evaluations demonstrate that information-aware advantage shaping is a powerful and general direction for token-efficient post-training. The code is available at https://github.com/YinhanHe123/IAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。