探究大模型是否在内部编码推理令牌的功能重要性。
Do LLMs Encode Functional Importance of Reasoning Tokens?
- 提出贪心剪枝法，按最小损失逐次删除推理令牌。
- 剪枝后模型在相同长度下表现优于前沿模型监督压缩基线。
- 发现注意力分数可预测剪枝顺序，表明模型有功能重要性结构。
大型语言模型通过生成长推理链解决复杂任务，但伴随计算成本上升和难以定位关键推理步骤的问题。现有压缩方法多依赖概率采样、启发式规则或前沿模型监督，却无法揭示模型是否内含令牌级功能重要性。本文提出一种保似然的贪心剪枝方法，迭代删除对模型似然影响最小的推理令牌，实现长度可控的推理链。在蒸馏框架中评估剪枝链效果，结果表明学生模型在相同推理长度下优于前沿模型监督压缩基线。进一步分析发现系统性剪枝模式，且注意力分数能有效预测剪枝优先级，表明模型确实对推理令牌构建了非平凡的功能重要性结构。
原文摘要 · Abstract (English)
Large language models solve complex tasks by generating long reasoning chains, achieving higher accuracy at the cost of increased computational cost and reduced ability to isolate functionally relevant reasoning. Prior work on compact reasoning shortens such chains through probabilistic sampling, heuristics, or supervision from frontier models, but offers limited insight into whether models internally encode token-level functional importance for answer generation. We address this gap diagnostically and propose greedy pruning, a likelihood-preserving deletion procedure that iteratively removes reasoning tokens whose removal minimally degrades model likelihood under a specified objective, yielding length-controlled reasoning chains. We evaluate pruned reasoning in a distillation framework and show that students trained on pruned chains outperform a frontier-model-supervised compression baseline at matched reasoning lengths. Finally, our analysis reveals systematic pruning patterns and shows that attention scores can predict greedy pruning ranks, further suggesting that models encode a nontrivial functional importance structure over reasoning tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。