熵值筛选无法有效压缩大模型推理链，信息分布于全程而非少数关键步骤。
Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
- 用熵值筛选推理步骤，效果不如随机删减。
- 仅数学题中低熵词元有效，因数字本身熵值低且含语义。
- 修复少量推理步骤的激活值即可恢复完整性能，说明信息分散分布。
熵基剪枝被提出用于压缩链式思维（CoT）推理，且精度损失可忽略。我们测试了不同模型和推理任务下低熵与高熵步骤选择方法的鲁棒性，发现熵值筛选在所有评估场景中均无优势，效果不优于随机剪枝。从句子转向词元层级后，我们发现仅在数学类基准上保留低熵词元有效，原因在于数值词元本身熵值较低，并携带语义信息。最后，通过修补少数几个词元的原始激活值，即可恢复接近完整的推理性能，提供了因果证据：任务信息并非集中于少数可通过启发式方法识别的词元，而是分布在整个推理链中。
原文摘要 · Abstract (English)
Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。