用强化学习动态调整大模型推理步骤，提速最多2.7倍。
Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
- 将推理出口选择建模为马尔可夫决策过程，通过离线强化学习优化策略。
- 在Llama-2和Llama-3上实现2.0~2.7倍加速，较静态基线快17%。
- 适合追求高效推理的部署场景，尤其对长文本生成有显著收益。
大语言模型的自回归推理速度缓慢。虽然自推测解码能加速这一过程，但其效率受固定退出层和推测长度等静态配置限制。本文将该优化问题重新建模为马尔可夫决策过程，提出LEDE框架，采用离线强化学习，根据生成序列的局部上下文动态选择最优退出层与推测长度，在计算成本与草稿质量间取得平衡。在Llama-2和Llama-3模型上的全面评估表明，LEDE相比自回归解码实现最高2.7倍的加速，相较于静态推测基线额外提升17%的效率。
原文摘要 · Abstract (English)
Large Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation lengths. We reframe this optimization as a \textbf{Markov Decision Process} and propose \textbf{LEDE}, a framework that uses offline reinforcement learning. LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context of the generated sequence at each step, balancing computational cost and draft quality. Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0\times$$\sim$$2.7\times$ speedup over autoregressive decoding and and provides an additional 17\% speedup over the static speculative baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。