通过内在重要性剪枝,让长推理链更短更快却不丢逻辑。
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning

- 基于模型自身注意力信号识别关键推理步骤
- 在数学数据集上减少50%-60%的令牌数且准确率不变
- 适合需要高效推理的复杂任务研究者
长链式思维(CoT)是近期推理模型成功的关键,但带来高计算开销和延迟。以往方法依赖外部压缩器,常与模型内部推理动态不匹配,导致关键逻辑步骤丢失。本文提出CRISP框架,通过挖掘模型内在重要性来压缩CoT。分析发现,推理终止标记[object Object]充当信息锚点,其注意力模式可有效区分必要推理与冗余内容。基于此,设计了利用内在注意力信号指导原子级压缩操作的策略。相比粗粒度剪枝,CRISP能有策略地提炼推理链,在保持逻辑连贯的同时最大化信息密度。在多种骨干模型和数学数据集上的实验证明,CRISP可实现50%-60%的令牌数缩减,且不牺牲准确性,有效缓解长上下文推理的效率瓶颈。代码已开源,以促进高效推理研究。
原文摘要 · Abstract (English)
Long Chain-of-Thought (CoT) reasoning is pivotal for the success of recent reasoning models but suffers from high computational overhead and latency. While prior works attempt to compress CoT via external compressor, they often fail to align with the model's internal reasoning dynamics, resulting in the loss of critical logical steps. This paper presents \textbf{C}ompressing \textbf{R}edundancy in Chain-of-Thought via \textbf{I}ntrinsic \textbf{S}aliency \textbf{P}runing (\textbf{CRISP}), a framework that compresses CoT by exploiting the model's intrinsic saliency. Our analysis reveals a distinct phenomenon: the reasoning termination token \texttt{[object Object]} acts as an information anchor, where its attention pattern effectively demarcates essential reasoning from redundancy. Based on this finding, we design a policy that utilizes these intrinsic attention signals to guide atomic compression operations. In contrast to coarse-grained pruning strategies, CRISP strategically distills the reasoning chain to maximize information density while preserving logical coherence. Empirical results across various backbone models and mathematical datasets demonstrate that CRISP achieves a 50-60% reduction in token count without compromising accuracy, effectively mitigating the efficiency bottleneck of long-context reasoning. We open-source our implementation to facilitate further research in efficient reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。