arXiv:2605.09490cs.CLcs.AR2026-05被引 1

让大模型推理时部分思考步骤存到内存,不丢数据也能省显存。

Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning

  • 按语义重要性分层管理思考过程,低重要性内容移至CPU内存
  • 仅3%内容永久删除,仍保持91%准确率(GSM8K)
  • 适合需要高精度推理的生产级大模型部署场景

推理型大模型生成数千个思维链标记,其键值缓存需驻留于稀缺的GPU显存(HBM)。当前主流方法是永久淘汰低重要性标记,但会导致准确率暴跌至0-2.5%。本文提出新思路:是否所有标记都必须在HBM中?我们设计了一种语义感知的内存层级结构,将标记分为四层——HBM、DDR、压缩存储和已淘汰。低重要性标记被移至CPU内存而非丢弃;在每次注意力计算前,以全精度预加载回显存,贡献等同于未离开显存的计算项。我们将其形式化为零近似误差卸载,并得出核心结论:准确率仅取决于永久淘汰比例,与留在HBM中的数量无关。在7B-32B三类模型、四个基准测试上,3×3网格实验验证该规律。仅3%淘汰率下,GSM8K准确率达91%,MATH-500达71%(n=200);14B模型在半显存占用下达到90%准确率(对比基线86%)。与当前最优方法R-KV相比,同等预算下本方法准确率提升至0-32%。真实系统原型显示数据迁移开销仅为5-7%,规模化分析预测可节省2-48GB HBM。

原文摘要 · Abstract (English)

Reasoning LLMs produce thousands of chain-of-thought tokens whose KV cache must reside in scarce GPU HBM. The dominant response -- permanently evicting low-importance tokens -- is catastrophic for reasoning: accuracy collapses to 0-2.5% when half the cache is removed. We ask a different question: must every token live in HBM, or can some live elsewhere? We introduce a semantics-aware memory hierarchy that sorts tokens into four tiers -- HBM, DDR, compressed, and evicted -- using cumulative attention scoring. Low-importance tokens are moved to CPU memory rather than destroyed; before each attention step they are prefetched back at full precision, contributing exactly the same terms as if they had never left the GPU. We formalize this as zero-approximation-error offloading and derive our central finding: accuracy depends solely on how many tokens are permanently discarded (the eviction ratio), not on how many remain in HBM. A controlled 3x3 grid over HBM and eviction ratios confirms this across three model scales (7B-32B) and four benchmarks. With only 3% eviction, the hierarchy retains 91% of full-cache accuracy on GSM8K and 71% on MATH-500 (n=200); at 14B scale it matches the uncompressed baseline (90% vs. 86%) while halving HBM occupancy. A head-to-head reproduction of R-KV -- the current SOTA eviction method -- on our setup achieves only 0-32% at comparable budgets. A system prototype with real GPU-CPU data movement shows that the price of this preservation is modest -- 5-7% transfer overhead -- and scaling analysis projects 2-48 GB HBM savings at production batch sizes.

大模型推理显存优化记忆层次语义分层

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。