动态树注意力提升大模型多头解码速度，不降质量。
Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention
- 用动态树结构替代固定树注意力，灵活生成候选序列。
- 实验显示速度提升，生成质量保持不变。
- 适合追求推理加速的LLM部署场景。
多头解码通过并行预测多个后续标记加速大语言模型（LLM）推理。它利用具有固定结构的树注意力生成并验证多个候选序列。本文将固定树注意力替换为动态树注意力，应用于MEDUSA框架。提出一种简单且低复杂度的策略来生成候选序列并构建动态树结构。初步实验表明，该方法在保持生成质量的同时提升了多头解码的推理效率，展示了候选生成优化在多头解码中的潜力。
原文摘要 · Abstract (English)
Multiple heads decoding accelerates the inference of Large Language Models (LLMs) by predicting next several tokens simultaneously. It generates and verifies multiple candidate sequences in parallel via tree attention with a fixed structure. In this paper, we replace the fixed tree attention with dynamic tree attention on multiple head decoding, specifically in the context of MEDUSA. We propose a simple and low complexity strategy to generate candidates and construct the dynamic tree structure. Preliminary experiments show that the proposed method improves the decoding efficiency of multiple head decoding for LLMs while maintaining the generation quality. This result demonstrates the potential for improvement of multiple head decoding in candidate generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。