让大模型在内存有限时也能处理超长文本。
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
- 用持续上下文蒸馏压缩关键信息，动态管理内存中的键值缓存。
- 在固定内存下处理超长序列，性能超越专为长文本训练的模型。
- 无需重新训练，适合移动端等资源受限场景使用。
长输入上下文处理仍是大语言模型（LLMs）在资源受限环境（如移动设备）中面临的重要挑战。本文提出 InfiniPot，一种新型的键值缓存控制框架，使预训练 LLM 能在固定内存约束下高效处理超长序列，且无需额外训练。InfiniPot 采用持续上下文蒸馏（CCD），通过新颖的重要性度量迭代压缩并保留关键信息，即使无法获取未来上下文也能有效维持重要数据。全面评估表明，InfiniPot 在多种自然语言处理任务中显著优于专为长上下文设计的模型，验证了其有效性与通用性。该工作推动了大模型在更广泛真实场景中的应用。
原文摘要 · Abstract (English)
Handling long input contexts remains a significant challenge for Large Language Models (LLMs), particularly in resource-constrained environments such as mobile devices. Our work aims to address this limitation by introducing InfiniPot, a novel KV cache control framework designed to enable pre-trained LLMs to manage extensive sequences within fixed memory constraints efficiently, without requiring additional training. InfiniPot leverages Continual Context Distillation (CCD), an iterative process that compresses and retains essential information through novel importance metrics, effectively maintaining critical data even without access to future context. Our comprehensive evaluations indicate that InfiniPot significantly outperforms models trained for long contexts in various NLP tasks, establishing its efficacy and versatility. This work represents a substantial advancement toward making LLMs applicable to a broader range of real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。