让语言模型自动跳过冗余注意力层，提速降耗不丢性能
Hopscotch: Discovering and Skipping Redundancies in Language Models
- 通过可训练缩放参数识别并跳过贡献小的注意力层
- 跳过4层后性能下降不足2%，保持输出质量稳定
- 无需改权重或预训练数据，兼容现有压缩技术
现代因果语言模型通过堆叠大量注意力模块提升性能，但并非所有模块对每个任务都必要。本文提出Hopscotch方法，能识别并跳过对任务贡献最小的注意力层，同时自适应调整剩余层的输出以保持质量。该方法引入轻量级可训练缩放参数，作用于注意力和MLP模块，缓解因移除注意力层导致的隐藏状态分布偏移。Hopscotch不修改模型权重，也不需访问预训练或指令微调数据，且与现有模型压缩技术兼容。在Llama-3.1-8B和Qwen2.5-7B上应用时，即使跳过4个注意力层，性能下降仍小于2%。
原文摘要 · Abstract (English)
Modern causal language models stack many attention blocks to improve performance, but not all blocks are necessary for every task. We propose Hopscotch, a simple yet effective method that identifies and skips attention blocks with least contributions to a task and adapts to preserve output quality. Hopscotch jointly optimizes which blocks to skip and how to scale the outputs of the remaining layers. By introducing lightweight, trainable scaling parameters to attention and MLP blocks, it mitigates distribution shifts in hidden states caused by removing attention blocks. Hopscotch does not modify model weights or require access to pretraining or instruction-tuning data, and is compatible with existing model compression techniques. When applied to $\texttt{Llama-3.1-8B}$ and $\texttt{Qwen2.5-7B}$, Hopscotch achieves less than a 2% drop in performance even after skipping four attention blocks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。