动态剪枝词汇表，让大模型更快更省地提前退出。
Dynamic Vocabulary Pruning in Early-Exit LLMs
- 在推理时动态剪掉无关词汇，减少计算量。
- 实测提升提前退出模型的效率，性能不降。
- 适合追求高效推理的大模型部署场景。
增大大型语言模型（LLMs）规模可提升性能，但会带来更慢、更昂贵的推理。提前退出是一种有前景的优化方法，可在中间层实现下一个词预测，从而提升推理效率。然而，现代模型庞大的词汇量使退出决策所需的置信度估计计算开销巨大，削弱了效率优势。为此，我们提出在测试时对词汇表进行动态剪枝：在初始层之一剪枝词汇，之后的前向传播全程使用缩小后的词汇表。实验表明，这种事后动态词汇剪枝能显著提升提前退出模型中置信度估计的效率，同时保持竞争力的性能。
原文摘要 · Abstract (English)
Increasing the size of large language models (LLMs) has been shown to lead to better performance. However, this comes at the cost of slower and more expensive inference. Early-exiting is a promising approach for improving the efficiency of LLM inference by enabling next token prediction at intermediate layers. Yet, the large vocabulary size in modern LLMs makes the confidence estimation required for exit decisions computationally expensive, diminishing the efficiency gains. To address this, we propose dynamically pruning the vocabulary at test time for each token. Specifically, the vocabulary is pruned at one of the initial layers, and the smaller vocabulary is then used throughout the rest of the forward pass. Our experiments demonstrate that such post-hoc dynamic vocabulary pruning improves the efficiency of confidence estimation in early-exit LLMs while maintaining competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。