通过自蒸馏与稀疏化，提升大模型生成速度且降低计算量。
SD$^2$: Self-Distilled Sparse Drafters
- 用自蒸馏和细粒度稀疏化训练高效对齐的草稿模型。
- 相比层剪枝模型，平均接受长度提升1.59倍，计算量减少超43.87%。
- 适合追求推理加速又不牺牲生成质量的工程应用者。
推测解码是降低大语言模型延迟的强大技术，提供容错框架以使用高度压缩的草稿模型。本文提出自蒸馏稀疏草稿模型（SD²），通过自数据蒸馏与细粒度权重稀疏化，生成高效且对齐良好的草稿模型。SD²系统性提升草稿词元接受率，显著降低乘加操作（MACs），即使在草稿模型与目标模型来自不同模型家族的通用辅助生成（UAG）场景下也表现优异。在Llama-3.1-70B目标模型上，相比层剪枝草稿模型，平均接受长度（MAL）提高1.59倍；相比密集草稿模型，MACs降低超过43.87%，仅牺牲8.36%的MAL。1.5B与3B的非结构化稀疏草稿模型在端到端延迟优化上优于密集与层剪枝模型，凸显稀疏感知微调与压缩策略在保持模型对齐的同时提升大模型推理效率的潜力。
原文摘要 · Abstract (English)
Speculative decoding is a powerful technique for reducing the latency of Large Language Models (LLMs), offering a fault-tolerant framework that enables the use of highly compressed draft models. In this work, we introduce Self-Distilled Sparse Drafters (SD$^2$), a novel methodology that leverages self-data distillation and fine-grained weight sparsity to produce highly efficient and well-aligned draft models. SD$^2$ systematically enhances draft token acceptance rates while significantly reducing Multiply-Accumulate operations (MACs), even in the Universal Assisted Generation (UAG) setting, where draft and target models originate from different model families. On a Llama-3.1-70B target model, SD$^2$ provides a 1.59$\times$ higher Mean Accepted Length (MAL) compared to layer-pruned draft models and reduces MACs by over 43.87% with a 8.36% reduction in MAL compared to a dense draft models. Our 1.5B and 3B unstructured sparse drafters outperform both dense and layer-pruned models in terms of end-to-end latency improvements; highlighting the potential of sparsity-aware fine-tuning and compression strategies to improve LLM inference efficiency while maintaining alignment with target models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。