用首个词意外度剪枝,让大模型推理更高效。
Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal
- 先用锚点保留核心结构,再基于首词意外度筛选关键步骤。
- 在多个基准上保持顶尖准确率,训练和推理成本大幅降低。
- 适合追求高效推理的模型部署场景,尤其擅长逻辑密集任务。
大型推理模型(LRM)通过扩展思维链(CoT)长度展现了强大能力,但过长的推理过程带来高昂的训练成本与推理延迟。现有压缩方法存在固有权衡:逐令牌方法常破坏语法与逻辑连贯性,基于困惑度的逐步方法因逻辑信息稀释而难以捕捉关键步骤。本文提出 ASAP(Anchor-guided, SurprisAl-based Pruning),一种从粗到精的 CoT 压缩框架。首先通过锚点引导剪枝保留核心推理结构,有效缩小后续处理搜索空间;基于‘逻辑分支集中于推理起始阶段’的洞察,引入新型首词意外度指标,实现逻辑感知剪枝,精准保留关键推理步骤;最后通过模型蒸馏,使模型在推理时能自主生成并利用这些简洁的 CoT。实验表明,ASAP 在多个基准上达到当前最优准确率,同时显著降低训练与推理开销。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities by scaling up the length of Chain-of-Thought (CoT). However, excessively long reasoning traces pose substantial challenges for training cost and inference latency. While various CoT compression approaches have emerged to address this challenge, they face inherent trade-offs: token-level methods often disrupt syntactic and logical coherence, while step-level methods based on perplexity fail to reliably capture the logically critical reasoning steps because of the dilution of logical information. In this paper, we propose ASAP (Anchor-guided, SurprisAl-based Pruning), a novel coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided pruning to preserve the core reasoning structure, which efficiently reduces the search space for subsequent processing. Leveraging the insight that logical branching choices are concentrated at the onset of reasoning steps, it then enables logic-aware pruning by selecting logically essential reasoning steps based on a novel first-token surprisal metric. Finally, ASAP distills the models to autonomously generate and leverage these concise CoTs at inference time, enabling efficient reasoning. Experiments show that ASAP achieves state-of-the-art accuracy across multiple benchmarks while substantially reducing training and inference costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。