利用显卡间高速互联，动态缓存大模型推理数据，提速超2倍。
Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference
- 通过显卡间直连内存做临时缓存，动态管理模型权重和注意力缓存
- 在真实场景下实现推理吞吐量提升超过2倍，关键组件访问速度显著加快
- 适合需要高吞吐大模型推理的云服务或高性能计算场景
大语言模型（LLM）推理日益受制于GPU内存容量而非计算吞吐量，主要因模型规模增长及自回归解码过程中键值（KV）缓存线性膨胀。现有方法通过将模型状态和KV张量卸载至主机内存缓解内存压力，但受限于有限的PCIe带宽导致显著延迟。我们提出Harvest，一种利用高带宽显卡间直连接口的机遇式GPU缓存管理框架，动态将模型权重与KV缓存放置于未使用的显卡内存中。Harvest将同级显卡内存视为临时缓存层，在保持正确性的前提下，有效降低动态内存环境下数据搬移开销。实验表明，通过Harvest加速两个常用推理组件——专家层权重与KV缓存条目的检索，可实现超过2倍的吞吐量提升。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the key-value (KV) cache during autoregressive decoding. Existing approaches mitigate memory pressure by offloading model state and KV tensors to host memory, but incur substantial latency due to limited PCIe bandwidth. We present Harvest, an opportunistic GPU cache management framework that exploits high-bandwidth peer-to-peer GPU interconnects to dynamically place model weights and KV cache in unused GPU memory. Harvest treats peer GPU memory as a transient cache tier, preserving correctness while reducing data movement overhead under dynamic memory availability. We demonstrate significant throughput speedup of more than 2 times by using Harvest to accelerate the retrieval of two widely-used inference components: expert layer weights and KV cache entries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。