自动分配不同任务给专用小模型，提升大模型推理速度与准确率
Automatic Task Detection and Heterogeneous LLM Speculative Decoding
- 根据任务类型自动分配到不同小模型，实现精准匹配
- 相比传统方法，推理速度最高提升2.64倍，准确率提高50%
- 适合需要多任务高效推理的场景，如智能客服、内容生成
推测解码通过结合草稿模型与目标模型，已成为加速大语言模型（LLM）推理的有效方法。然而，现有方法常因草稿模型能力有限，在下游任务中面临接受率与解码速度之间的权衡，难以在多样化任务中保证效率。为此，我们提出一种面向下游任务优化的推测解码算法，包含自动任务划分与分配机制，能将下游任务自动分类为多个子任务，并分配至一组异构草稿模型。每个草稿模型使用特定任务数据与目标模型对齐，从而提升推理结果一致性。此外，该方法引入轻量级在线提示分类器，动态将提示路由至合适的草稿模型。实验表明，所提方法在草稿准确率上比基线提升6%至50%，同时在LLM推理中实现1.10x至2.64x的速度提升。
原文摘要 · Abstract (English)
Speculative decoding, which combines a draft model with a target model, has emerged as an effective approach to accelerate large language model (LLM) inference. However, existing methods often face a trade-off between the acceptance rate and decoding speed in downstream tasks due to the limited capacity of the draft model, making it difficult to ensure efficiency across diverse tasks. To address this problem, we propose a speculative decoding algorithm tailored for downstream task optimization. It includes an automatic task partitioning and assigning method, which automatically categorizes downstream tasks into different sub-tasks and assigns them to a set of heterogeneous draft models. Each draft model is aligned with the target model using task-specific data, thereby enhancing the consistency of inference results. In addition, our proposed method incorporates an online lightweight prompt classifier to dynamically route prompts to the appropriate draft model. Experimental results demonstrate that the proposed method improves draft accuracy by 6% to 50% over vanilla speculative decoding, while achieving a speedup of 1.10x to 2.64x in LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。