Akashic用记忆分块技术提升大模型推理效率与准确率
Akashic: A Low-Overhead LLM Inference Service with MemAttention
- 将上下文分块并建模块间语义关系，避免重复加载完整历史
- 在四个任务中最高提升10.2%准确率，吞吐量提高1.21倍
- 适合需要长期记忆的AI助手、多轮对话系统使用
基于大模型的智能体系统在多轮交互、工具调用和跨会话流程中持续积累上下文。每次请求都重播完整历史很快变得不切实际：长上下文增加预填充开销，可能超出上下文限制，且常使任务相关证据被无关内容淹没，降低服务效率与输出质量。我们提出Akashic，一种以MemAttention为核心的低开销记忆系统，将上下文组织为有限大小的块，并建模块间语义关系，在不重复写入完整历史的前提下保留跨块证据。Akashic进一步采用软硬件协同设计的内存布局策略，将可能共同检索的块就近放置，减少检索碎片化和I/O开销。在四个代表性工作负载和三种模型规模下，Akashic相比强基准内存方案，任务准确率最高提升10.2点，吞吐量最高提升1.21倍，可持续请求率最高提升1.88倍。
原文摘要 · Abstract (English)
Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。