通过预测未来隐状态，实现LLM一次生成多个词元，大幅降低推理延迟。
FIRP: Faster LLM inference via future intermediate representation prediction
- 用线性变换预测未来词元的中间隐藏状态，用于提前解码。
- 在多个模型和数据集上实现1.9x–3x的推理加速。
- 适合追求高效推理的部署场景，尤其适用于高延迟敏感应用。
大型语言模型（LLMs）在众多任务中表现出色，但其自回归解码方式每次仅生成一个词元，未能充分利用GPU的并行计算能力，导致显著延迟。为解决此问题，我们提出一种名为FIRP的新颖推测解码方法，可在每一步解码中生成多个词元。该方法通过预测未来词元的中间隐藏状态（尚未解码），并利用这些伪隐藏状态进行未来词元的解码；其中，伪隐藏状态通过LLM中间层的简单线性变换获得。一旦预测，它们参与后续所有层的计算，从而融入更丰富的语义信息。随着网络层数加深，伪隐藏状态与真实隐藏状态之间的语义差距逐渐缩小，使得未来词元的高精度解码成为可能。大量实验验证了FIRP的有效性，在多个模型和数据集上实现了1.9x–3x的加速比，分析实验也证实了其设计动机。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have shown remarkable performance across a wide range of tasks. Despite this, the auto-regressive nature of LLM decoding, which generates only a single token per forward propagation, fails to fully exploit the parallel computational power of GPUs, leading to considerable latency. To address this, we introduce a novel speculative decoding method named FIRP which generates multiple tokens instead of one at each decoding step. We achieve this by predicting the intermediate hidden states of future tokens (tokens have not been decoded yet) and then using these pseudo hidden states to decode future tokens, specifically, these pseudo hidden states are predicted with simple linear transformation in intermediate layers of LLMs. Once predicted, they participate in the computation of all the following layers, thereby assimilating richer semantic information. As the layers go deeper, the semantic gap between pseudo and real hidden states is narrowed and it becomes feasible to decode future tokens with high accuracy. To validate the effectiveness of FIRP, we conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets, analytical experiments also prove our motivations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。