提出CORAL框架，让大模型推理更准更快
CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter
- 跨步表征对齐提升多步训练一致性
- 推理速度提升2.5倍至4.07倍
- 适合需要高效推理的大模型部署
推测解码是一种通过轻量级推测稿模型加速大语言模型（LLM）推理的强大技术。然而，现有方法因训练与推理阶段不一致而性能受限。近期方法虽采用多步训练策略，但不同训练步骤的复杂输入导致稿模型难以收敛。为此，我们提出CORAL框架，通过引入跨步表征对齐机制，显著提升推测解码性能。同时，我们发现语言模型头是稿模型推理速度的主要瓶颈，提出参数分组机制，在推理时仅激活部分头参数，大幅降低延迟。我们在三种LLM族和三个基准数据集上评估CORAL，速度提升达2.50x-4.07x，优于EAGLE-2和HASS等先进方法。结果表明，CORAL有效缓解了训练-推理不一致问题，为大规模词汇量的现代LLM提供了显著加速。
原文摘要 · Abstract (English)
Speculative decoding is a powerful technique that accelerates Large Language Model (LLM) inference by leveraging a lightweight speculative draft model. However, existing designs suffers in performance due to misalignment between training and inference. Recent methods have tried to solve this issue by adopting a multi-step training strategy, but the complex inputs of different training steps make it harder for the draft model to converge. To address this, we propose CORAL, a novel framework that improves both accuracy and efficiency in speculative drafting. CORAL introduces Cross-Step Representation Alignment, a method that enhances consistency across multiple training steps, significantly improving speculative drafting performance. Additionally, we identify the LM head as a major bottleneck in the inference speed of the draft model. We introduce a weight-grouping mechanism that selectively activates a subset of LM head parameters during inference, substantially reducing the latency of the draft model. We evaluate CORAL on three LLM families and three benchmark datasets, achieving speedup ratios of 2.50x-4.07x, outperforming state-of-the-art methods such as EAGLE-2 and HASS. Our results demonstrate that CORAL effectively mitigates training-inference misalignment and delivers significant speedup for modern LLMs with large vocabularies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。