用注意力机制剔除冗余推理路径，提升效率与准确率
FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning
- 基于注意力权重识别并移除低质量推理节点
- 减少69.7%的令牌消耗，准确率提升26.7%
- 适合追求高效推理的模型部署场景
我们提出FROST，一种基于注意力的高效推理方法。不同于传统方法，FROST利用注意力权重剪枝非关键推理路径，生成更短且更可靠的推理轨迹。方法上，引入推理异常值概念，并设计基于注意力的剔除机制。理论上，FROST在保留并增强模型推理能力的同时，实现句子级异常值消除。实证上，在四个基准测试中使用两个强推理模型（Phi-4-Reasoning 和 GPT-OSS-20B）验证，优于TALE和ThinkLess等前沿方法。显著地，FROST平均减少69.68%的令牌使用量，相比基线模型准确率提升26.70%。在注意力异常值指标评估中，最大无穷范数降低15.97%，平均峰度下降91.09%。代码已公开于https://github.com/robinzixuan/FROST。
原文摘要 · Abstract (English)
We propose FROST, an attention-aware method for efficient reasoning. Unlike traditional approaches, FROST leverages attention weights to prune uncritical reasoning paths, yielding shorter and more reliable reasoning trajectories. Methodologically, we introduce the concept of reasoning outliers and design an attention-based mechanism to remove them. Theoretically, FROST preserves and enhances the model's reasoning capacity while eliminating outliers at the sentence level. Empirically, we validate FROST on four benchmarks using two strong reasoning models (Phi-4-Reasoning and GPT-OSS-20B), outperforming state-of-the-art methods such as TALE and ThinkLess. Notably, FROST achieves an average 69.68% reduction in token usage and a 26.70% improvement in accuracy over the base model. Furthermore, in evaluations of attention outlier metrics, FROST reduces the maximum infinity norm by 15.97% and the average kurtosis by 91.09% compared to the base model. Code is available at https://github.com/robinzixuan/FROST
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。