TokenSwift让超长文本生成提速3倍以上，不损失质量。
TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation
- 通过优化模型重载、KV缓存管理和重复生成问题，实现无损加速。
- 在1.5B到14B模型上，生成速度提升超3倍，支持10万词元序列。
- 适合需要高效生成超长文本的科研与工业场景。
大语言模型生成超长序列（最长可达10万词元）虽日益重要，但耗时严重。传统推测解码方法直接扩展生成长度无法提速，甚至有害。我们深入分析发现三大瓶颈：频繁模型重载、动态键值（KV）管理困难和重复生成。为此提出TokenSwift框架，显著加速超长序列生成，同时保持目标模型原有质量。实验表明，该框架在1.5B、7B、8B、14B不同规模与架构（MHA、GQA）模型上均实现超过3倍的加速，大幅节省生成时间，为前所未有的长序列生成提供可扩展、高效的解决方案。代码见https://github.com/bigai-nlco/TokenSwift。
原文摘要 · Abstract (English)
Generating ultra-long sequences with large language models (LLMs) has become increasingly crucial but remains a highly time-intensive task, particularly for sequences up to 100K tokens. While traditional speculative decoding methods exist, simply extending their generation limits fails to accelerate the process and can be detrimental. Through an in-depth analysis, we identify three major challenges hindering efficient generation: frequent model reloading, dynamic key-value (KV) management and repetitive generation. To address these issues, we introduce TOKENSWIFT, a novel framework designed to substantially accelerate the generation process of ultra-long sequences while maintaining the target model's inherent quality. Experimental results demonstrate that TOKENSWIFT achieves over 3 times speedup across models of varying scales (1.5B, 7B, 8B, 14B) and architectures (MHA, GQA). This acceleration translates to hours of time savings for ultra-long sequence generation, establishing TOKENSWIFT as a scalable and effective solution at unprecedented lengths. Code can be found at https://github.com/bigai-nlco/TokenSwift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。