用分层记忆机制优化长文本推理，显著降低延迟和内存压力。
TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
- 将KV缓存按时间分层，近的放快存、远的放慢存，模拟人脑记忆规律。
- 在128K上下文任务中，跨层通信量减少5.94倍，延迟降低76%。
- 适合需要高效长文本处理的部署场景，如超长文档生成与分析。
键值（KV）缓存对大语言模型高效推理至关重要，但其内存占用随上下文长度线性增长，造成严重可扩展瓶颈。现有方法通常将所有KV状态视为同等重要，隐含假设精度和访问性一致。然而，这与人类记忆系统相悖——记忆的清晰度、回忆频率和相关性随时间接近度变化。受此启发，我们提出TTKV，一种将人类记忆机制映射到KV缓存的管理框架。TTKV将KV缓存划分为具有异构容量和精度的时间层级：(1) 层级布局，通过HBM和DRAM分离快速与慢速存储；(2) 层级内容，根据时间临近度将更近期的KV状态分配至更快、更高精度层级；(3) 层级交互，采用块级流式注意力，在访问慢速层级时重叠通信与计算。实验表明，TTKV在128K上下文任务中使跨层级通信量减少5.94倍，延迟最高降低76%，吞吐量提升2倍，优于强基线。
原文摘要 · Abstract (English)
Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。