通过分层缓存优化生成式推荐模型推理，显著提升效率
MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches
- 用主机内存虚拟化GPU显存，实现大规模用户状态存储
- 在真实数据集上提速达3.1倍，缓存命中率超98.5%
- 适合高并发生成式推荐系统部署与工业级服务场景
生成式推荐模型虽具备强大建模能力，但因需反复编码长用户历史而带来高昂的推理开销。尽管跨请求键值（KV）缓存复用具有巨大优化潜力，但单个用户状态规模庞大，导致存储爆炸，远超物理GPU容量。我们提出MTServe，一种分层缓存管理系统，通过利用主机内存作为可扩展的后备存储，虚拟化GPU内存。为弥合层级间I/O差距，MTServe引入一系列系统级优化，包括混合存储布局、异步数据传输流水线和基于局部性的替换策略。在公开及生产数据集上，MTServe实现了最高3.1倍的加速，同时保持近完美的缓存命中率（>98.5%）。
原文摘要 · Abstract (English)
Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。