发现推理链中冗余步骤被忽略,提出新方法有效压缩且不丢准确率。
Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

- 基于信息论设计无训练度量CAID,识别低效推理步骤
- 在GSM8K等数据集上实现31%-53%的令牌压缩率
- 适合需要高效推理的模型部署与优化场景
Chain-of-Thought(CoT)提示显著提升了大语言模型的推理能力,但常因过度推理导致计算成本高昂:生成冗余、冗长或无关的步骤。现有推理步骤评估器虽能检测逻辑错误和事实错误,但无法惩罚那些虽有效却低效的推理步骤,这些步骤增加了令牌使用却无助于解题。为系统诊断此问题,我们引入RIV-GSM8K基准,包含五类效率缺陷,如循环推理和过度分解。实验表明,当前先进评估器难以区分这些低效步骤与必要推理。为此,我们提出CAID(上下文感知信息密度),一种无需训练的信息理论度量,可识别低效步骤。为验证其实际价值,我们将CAID用于PACE后处理压缩策略。控制实验显示,其增益非来自简单剪枝:相比随机移除和基于PRM的基线,它在更高压缩率下仍保持准确率。在GSM8K、StrategyQA和ARC-Challenge上的结果表明,PACE将令牌消耗减少31%-53%,同时维持准确率,证明CAID能有效去除推理链中的信息泡沫而不损害推理有效性。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps. While existing reasoning step evaluators effectively detect logical fallacies and factual errors, our analysis reveals a critical blind spot: they fail to penalize valid but inefficient reasoning steps that inflate token usage without contributing to the solution. To systematically diagnose this limitation, we introduce RIV-GSM8K, a diagnostic benchmark injected with five distinct types of inefficiencies, including circular reasoning and excessive decomposition. Diagnostic experiments reveal that state-of-the-art evaluators struggle to distinguish these inefficiencies from necessary reasoning. To address this gap, we propose CAID (Context-Aware Information Density), a training-free metric grounded in information theory that identifies low-utility steps. To validate the metric's practical utility, we apply it within PACE, a post-hoc compression strategy. Additional control experiments show that the gains of PACE are not explained by trivial pruning: compared with random step removal and PRM-based compression baselines, it preserves accuracy at substantially higher compression rates. Empirical results on GSM8K, StrategyQA, and ARC-Challenge demonstrate that PACE reduces token consumption by 31-53% while maintaining accuracy, confirming that CAID successfully distills informational froth from reasoning chains without compromising deductive validity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。