动态跳过Transformer层，按需分配计算量，提升大模型生成效率。
Adaptive Layer-skipping in Pre-trained LLMs
- 引入可插拔路由模块，动态决定每步生成用多少层。
- 在Llama-3-8B上跳过8层仍保持完整性能，节省大量计算。
- 适合追求推理效率优化的研究者与工程实践者。
现有层跳过方法旨在加速大语言模型（LLM）的文本生成，但对生成过程中不同标记的计算需求变化关注不足。本文提出FlexiDepth，一种通过插入式路由器与适配器实现动态计算调整的方法，无需修改原始参数即可自适应调整Transformer层数。应用于Llama-3-8B时，可跳过32层中的8层，同时维持全基准性能。实验表明，不同类型的标记所需计算量差异显著：重复性或固定短语生成只需较少层数，而涉及计算或高不确定性的标记则需更多层数。尽管计算量大幅减少，但由于跳过模式不均和输入输出开销，当前尚未实现实际运行时间加速。为推动后续研究，作者开源了FlexiDepth及记录层分配模式的数据集。
原文摘要 · Abstract (English)
Various layer-skipping methods have been proposed to accelerate token generation in large language models (LLMs). However, limited attention has been paid to a fundamental question: How do computational demands vary across the generation of different tokens? In this work, we introduce FlexiDepth, a method that dynamically adjusts the number of Transformer layers used in text generation. By incorporating a plug-in router and adapter, FlexiDepth enables adaptive computation in LLMs without modifying their original parameters. Applied to Llama-3-8B, it skips 8 out of 32 layers while maintaining full benchmark performance. Our experiments reveal that computational demands in LLMs significantly vary based on token type. Specifically, generating repetitive tokens or fixed phrases requires fewer layers, whereas producing tokens involving computation or high uncertainty requires more layers. Despite the computational savings, FlexiDepth does not yet achieve wall-clock speedup due to varied skipping patterns and I/O overhead. To inspire future work and advance research on practical speedup, we open-sourced FlexiDepth and a dataset documenting its layer allocation patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。