通过动态选择层数，用23.3%的层达到全模型效果。
Dynamic layer selection in decoder-only transformers
- 按序列动态分配计算量，而非逐词调整。
- 仅用23.3%的层数即可保持全模型性能。
- 适合追求推理效率的LLM部署场景。
大型语言模型（LLMs）的规模促使人们寻求推理优化。动态推理可根据输入样本调整结构以降低计算成本。我们实证考察了自然语言生成中两种常见方法：层跳过与提前退出。发现预训练的仅解码器模型对层跳过具有更强鲁棒性，而基于隐藏状态的逐词计算自适应难以实现。此外，我们构建了一个理想控制器，证明按序列动态分配计算量可带来显著效率提升。惊人的是，存在一种分配策略，在平均仅使用23.3%的层数时，仍能获得与完整模型相当的性能。
原文摘要 · Abstract (English)
The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。