让不同词汇表的模型也能协同加速推理，无需重新训练。
TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs
- 用动态时间规整算法对齐不同模型的词元序列
- 在多个任务上实现1.57倍推理加速
- 可直接使用现成模型，无需修改或重训
加速大语言模型（LLM）推理是生成式AI的关键挑战。推测解码（SD）显著提升了推理效率，但其应用受限于一个根本问题：草案模型与目标模型必须共享相同词汇表，这限制了可用草案模型的范围，且常需从头训练新模型。受经典动态时间规整（DTW）算法启发，我们提出TokenTiming方法，通过重新编码草案词元序列生成新的目标词元序列，并利用DTW建立映射关系，将概率分布迁移用于推测采样。该方法可兼容不同词汇表，支持任意现成模型，无需重新训练或修改。我们在多种任务上进行了全面实验，实现了1.57倍的加速。本工作实现了通用推测解码，使SD成为更灵活、实用的LLM加速工具。
原文摘要 · Abstract (English)
Accelerating the inference of large language models (LLMs) has been a critical challenge in generative AI. Speculative decoding (SD) substantially improves LLM inference efficiency. However, its utility is limited by a fundamental constraint: the draft and target models must share the same vocabulary, thus limiting the herd of available draft models and often necessitating the training of a new model from scratch. Inspired by Dynamic Time Warping (DTW), a classic algorithm for aligning time series, we propose the algorithm TokenTiming for universal speculative decoding. It operates by re-encoding the draft token sequence to get a new target token sequence, and then uses DTW to build a mapping to transfer the probability distributions for speculative sampling. Benefiting from this, our method accommodates mismatched vocabularies and works with any off-the-shelf models without retraining and modification. We conduct comprehensive experiments on various tasks, demonstrating 1.57x speedup. This work enables a universal approach for draft model selection, making SD a more versatile and practical tool for LLM acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。