无需微调，按上下文动态生成草稿，加速大模型推理。
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
- 根据输入上下文动态生成草稿模型，不依赖固定预训练草稿。
- 在自推测解码中表现媲美当前最先进方法。
- 即插即用，轻量级，适合快速部署到现有系统。
我们提出一种简单高效的实时推理加速方法，用于大型语言模型。与其它（自）推测解码技术不同，该方法无需微调或黑箱优化来生成固定草稿模型，而是基于简单规则动态生成适应输入上下文的草稿模型。实验证明，该轻量级算法在自推测解码任务上性能可与当前最先进方法媲美，同时具备真正即插即用的特性，无需额外训练或复杂配置。
原文摘要 · Abstract (English)
We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or black-box optimization to generate a fixed draft model, relying instead on simple rules to generate varying draft models adapted to the input context. We show empirically that our light-weight algorithm is competitive with the current SOTA for self-speculative decoding, while being a truly plug-and-play method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。