让边缘设备无限处理长文本，内存占用更低
EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices
- 用可训练的门控模块压缩存储，动态控制记忆使用
- 长文本任务性能接近基线，内存和首字延迟显著优化
- 兼容标准Transformer，仅需微调少量参数
基于Transformer的大语言模型在边缘设备上处理长序列时面临注意力机制二次复杂度和键值（KV）缓存内存增长的挑战。现有KV缓存优化在长输出任务中难以避免令牌淘汰，而替代序列建模架构又难以融入现有Transformer体系。本文提出EdgeInfinite，通过可训练的记忆门控模块将压缩记忆集成到基于Transformer的LLM中，保持与标准Transformer架构完全兼容，仅需微调少量参数，并支持长短上下文任务的智能路由。实验表明，EdgeInfinite在长上下文基准测试中表现接近基线模型，同时显著优化内存消耗和首个输出时间。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) encounter challenges in processing long sequences on edge devices due to the quadratic complexity of attention mechanisms and growing memory demands from Key-Value (KV) cache. Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks, while alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructure. We present EdgeInfinite, a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. This approach maintains full compatibility with standard Transformer architectures, requiring fine-tuning only a small part of parameters, and enables selective activation of the memory-gating module for long and short context task routing. The experimental result shows that EdgeInfinite achieves comparable performance to baseline Transformer-based LLM on long context benchmarks while optimizing memory consumption and time to first token.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。