发现大模型无需额外设计就能提前退出，显著加速推理
Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
- 不加新层也不联合优化，直接利用模型内部自然能力实现提前输出
- 在LLaMA模型上验证，提前退出可节省40%以上计算量且准确率损失小于2%
- 适合追求高效推理的部署场景，尤其对资源受限设备有重要意义
大型语言模型（LLMs）在各类下游任务中表现卓越，但其庞大的参数导致推理速度缓慢。早退（Early Exit, EE）通过从中间层生成输出而非使用整个模型，为加速自回归解码提供了可行方案。然而，传统方法需增加输出层并进行联合优化，限制了其在LLMs中的应用。本文研究了无需额外输出层和联合优化的EE可行性。结果表明，EE是基于Transformer模型的天然能力。虽然联合优化本身不能赋予模型早退能力，但能通过门控函数提升最优早退层定位精度。此外，我们基于LLaMA模型从子词层面揭示了早退行为模式，并探索了基于子层的早退潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit exceptional performance across various downstream tasks. However, they encounter limitations due to slow inference speeds stemming from their extensive parameters. The early exit (EE) is an approach that aims to accelerate auto-regressive decoding. EE generates outputs from intermediate layers instead of using the whole model, which offers a promising solution to this challenge. However, additional output layers and joint optimization used in conventional EE hinder the application of EE in LLMs. In this paper, we explore the possibility of LLMs EE without additional output layers and joint optimization. Our findings indicate that EE is a natural capability within transformer-based models. While joint optimization does not give model EE capability, it must be employed to address challenges by improving the accuracy of locating the optimal EE layer through gating functions. Additionally, our study reveals patterns in EE behavior from a sub-word perspective based on the LLaMA model and the potential possibility for EE based on sub-layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。