通过键值相似性在线剪枝，加速大模型推理且无需校准数据。
Leveraging KV Similarity for Online Structured Pruning in LLMs
- 基于注意力头间键值相似性，动态判断冗余token并跳过计算。
- 在50%剪枝率下仍保持MMLU等难题的强性能，准确率损失小于1.5%。
- 无额外内存开销，适合部署在资源受限的实时推理场景。
剪枝已成为加速大语言模型（LLM）推理的有前景方向，但现有方法常因依赖无法泛化的离线校准数据而出现不稳定问题。本文提出Token Filtering，一种轻量级在线结构化剪枝技术，在推理过程中直接做出剪枝决策，无需任何校准数据。核心思想是通过联合键值相似性衡量令牌冗余，跳过冗余注意力计算，从而降低推理开销并保留关键信息。为进一步提升稳定性，设计了方差感知融合策略，自适应加权各注意力头的键值相似性，确保在高剪枝率下仍保留重要令牌。该设计不引入额外内存开销，提供更可靠的令牌重要性判据。在LLaMA-2（7B/13B）、LLaMA-3（8B）和Mistral（7B）上的大量实验表明，Token Filtering持续优于先前结构化剪枝方法，在常识推理基准上保持准确率，并在50%剪枝率下仍对MMLU等挑战性任务保持强性能。
原文摘要 · Abstract (English)
Pruning has emerged as a promising direction for accelerating large language model (LLM) inference, yet existing approaches often suffer from instability because they rely on offline calibration data that may not generalize across inputs. In this work, we introduce Token Filtering, a lightweight online structured pruning technique that makes pruning decisions directly during inference without any calibration data. The key idea is to measure token redundancy via joint key-value similarity and skip redundant attention computations, thereby reducing inference cost while preserving critical information. To further enhance stability, we design a variance-aware fusion strategy that adaptively weights key and value similarity across heads, ensuring that informative tokens are retained even under high pruning ratios. This design introduces no additional memory overhead and provides a more reliable criterion for token importance. Extensive experiments on LLaMA-2 (7B/13B), LLaMA-3 (8B), and Mistral (7B) demonstrate that Token Filtering consistently outperforms prior structured pruning methods, preserving accuracy on commonsense reasoning benchmarks and maintaining strong performance on challenging tasks such as MMLU, even with 50% pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。