arXiv:2605.04450cs.DCcs.IR2026-05中稿 · SC 2026

动态调整推荐系统内存分配,显著降低推理延迟。

When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving

论文配图:When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving
图 1 · 摘自论文原文
  • 用强化学习动态调节嵌入与键值缓存的显存分配比例。
  • 实测将99%请求延迟降低24%-38%,满足服务等级承诺。
  • 适合大规模生成式推荐系统部署与优化场景。

生成式推荐推理中,嵌入热缓存(EMB)与键值缓存(KV)在有限的GPU HBM内存中直接竞争:分配更多内存给一方会提升其效率,但损害另一方性能。现有系统分别优化两者,忽略了最优的EMB-KV分配比在不同工作负载下可相差高达0.35,导致20-30%的延迟改进未被实现。虽然在线重分配能弥补这一差距,但传统方法会在关键路径引入主机到设备的数据回传,造成99%延迟超标。为此,我们提出RACER,通过两个核心组件在运行时联合管理显存分配与请求调度:(1) 自适应内存分配,采用三层PPO控制器(冻结基础策略、在线残差适配器、突发感知恢复控制器),实现32μs决策延迟,且分配比始终维持在离线最优值的0.024-0.029误差范围内;(2) EMB-KV感知调度,综合考虑键值驻留性、嵌入局部性和节点负载,避免异构分配下的调度低效。在32节点A100集群上对三个生产规模数据集的评估显示,RACER相比最佳静态策略将P99延迟降低24-38%,在平稳、趋势和突发三种负载下均达到93.5-99.6%的服务等级承诺满足率,显著优于现有最先进基线,且不牺牲吞吐量。

原文摘要 · Abstract (English)

Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficiency but degrades the other. Existing systems optimize them in isolation, overlooking that the optimal EMB-KV allocation ratio can shift by up to 0.35 across workload regimes, leaving 20-30\% latency improvement unrealized. While online reallocation is required to close this gap, naive approaches introduce H2D refill traffic on the critical path, causing P99 SLO violations. To address this, we present RACER, which jointly manages HBM allocation and request routing at runtime through two key components: (1) Adaptive Memory Allocation, a three-layer PPO-based controller (frozen base policy, online residual adapter, and burst-aware recovery controller) that achieves $32\,\mathrm{μs}$ decision latency while staying within 0.024-0.029 of the offline-optimal ratio; and (2) EMB-KV-Aware Scheduling, which routes requests by jointly considering KV residency, embedding locality, and node load to avoid routing inefficiencies under heterogeneous allocations. Evaluations on three production-scale datasets over a 32-node A100 cluster show that RACER reduces P99 latency by 24-38\% over the best static policy and achieves 93.5-99.6\% SLO satisfaction across Steady, Trend, and Burst workloads, significantly outperforming state-of-the-art baselines without sacrificing throughput.

推荐系统内存优化生成模型动态调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。