无需训练，通过跳层压缩模型加速大模型生成
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
- 利用动态规划跳过验证模型中间层，构建无额外训练的草稿模型
- 在LLaMA3上实现1.3倍至1.7倍的生成加速，保持原文分布不变
- 即插即用，适合追求推理加速且不想改模型的开发者
自回归生成中，推测解码（SD）是一种提升大型语言模型（LLM）推理效率的有前景方法。其性能主要依赖草稿模型与验证模型的一致性。然而，现有方法通常需要额外训练模块，实现复杂且难以兼容不同模型。本文提出CLaSp，一种基于上下文的层跳过策略，用于自推测解码。与以往方法不同，CLaSp无需额外模块或训练，而是通过跳过验证模型的中间层，构建压缩版草稿模型，实现即插即用。我们设计了一种动态规划算法，利用上一验证阶段的完整隐藏状态作为目标，优化层跳过过程，使策略能每轮验证后动态调整，无需预设跳层集合。实验表明，该方法在多个下游任务中对LLaMA3系列模型实现了1.3倍至1.7倍的加速，同时不改变原始生成文本的分布。
原文摘要 · Abstract (English)
Speculative decoding (SD) is a promising method for accelerating the decoding process of Large Language Models (LLMs). The efficiency of SD primarily hinges on the consistency between the draft model and the verify model. However, existing drafting approaches typically require additional modules to be trained, which can be challenging to implement and ensure compatibility across various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping strategy for self-speculative decoding. Unlike prior methods, CLaSp does not require additional drafting modules or extra training. Instead, it employs a plug-and-play mechanism by skipping intermediate layers of the verify model to construct a compressed draft model. Specifically, we develop a dynamic programming algorithm that optimizes the layer-skipping process by leveraging the complete hidden states from the last verification stage as an objective. This enables CLaSp to dynamically adjust its layer-skipping strategy after each verification stage, without relying on pre-optimized sets of skipped layers. Experimental results across diverse downstream tasks demonstrate that CLaSp achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the original distribution of the generated text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。