让大模型单卡处理300万字上下文，速度提升18倍
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
- 动态分层删减无关文本，只留关键信息
- 单卡处理300万token，速度比原来快18.95倍
- 无需重训练，适合长文档生成与分析
现代大语言模型在处理超长上下文时面临推理慢、显存高的难题，且多数预训练模型无法泛化到超过原始训练长度的序列。为实现高效实用的长上下文处理，我们提出InfiniteHiP框架，通过模块化分层令牌剪枝算法动态删除无关上下文，加速推理；同时根据模型内部注意力模式选择性应用不同RoPE调整方法，实现对更长序列的泛化能力；此外，将键值缓存卸载至主机内存，显著降低GPU显存压力。结果表明，InfiniteHiP可在单张L40s 48GB GPU上处理高达300万token的上下文（较以往扩大3倍），且无永久信息丢失。该框架在100万token上下文中实现18.95倍的注意力解码加速，无需额外训练。我们在SGLang框架中实现并验证了其有效性与实用性。
原文摘要 · Abstract (English)
In modern large language models (LLMs), handling very long context lengths presents significant challenges as it causes slower inference speeds and increased memory costs. Additionally, most existing pre-trained LLMs fail to generalize beyond their original training sequence lengths. To enable efficient and practical long-context utilization, we introduce InfiniteHiP, a novel, and practical LLM inference framework that accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm. Our method also allows generalization to longer sequences by selectively applying various RoPE adjustment methods according to the internal attention patterns within LLMs. Furthermore, we offload the key-value cache to host memory during inference, significantly reducing GPU memory pressure. As a result, InfiniteHiP enables the processing of up to 3 million tokens on a single L40s 48GB GPU -- 3x larger -- without any permanent loss of context information. Our framework achieves an 18.95x speedup in attention decoding for a 1 million token context without requiring additional training. We implement our method in the SGLang framework and demonstrate its effectiveness and practicality through extensive evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。