用学习增强方法提升GPU推理缓存效率,兼顾性能与可靠性。
Toward Robust and Efficient ML-Based GPU Caching for Modern Inference
- 结合预测与经典算法,实现精准与鲁棒的缓存决策
- 在大模型任务中降低28.3%的首令牌延迟,提升24.2%吞吐量
- 适合对缓存性能要求高且部署成本敏感的生产系统
现代GPU推理中,缓存效率仍是主要瓶颈,传统启发式策略如LRU可能远逊于离线最优。现有基于学习的缓存系统主要通过改进预测器来提升命中率,但常盲目依赖预测结果,导致预测不准时性能不可靠。相比之下,新兴的学习增强缓存算法通过谨慎融合预测信息,实现了性能保证:在预测准确时接近最优(1-一致性),预测错误时仍保持有界最坏性能(O(k)-鲁棒性)。然而实际部署仍面临挑战。理想算法需满足严格的时间与空间效率要求,且部署开销低。本文提出学习增强型LRU,具备1-一致性和O(k)-鲁棒性,时间与空间开销低,兼容性强。在此基础上构建了名为LCR的GPU缓存系统,将理论优势转化为实际性能。实验表明,LCR在大模型工作负载下可降低高达28.3%的P99首令牌延迟(TTFT),在深度学习推荐模型(DLRM)工作负载下提升最高24.2%吞吐量。即使预测质量差,性能也仅平滑下降,接近传统LRU表现,体现实用级鲁棒性。
原文摘要 · Abstract (English)
In modern GPU inference, cache efficiency remains a major bottleneck, and heuristic policies such as \textsc{LRU} can perform far worse than the offline optimum. Existing learning-based caching systems improve hit rates mainly through predictor design, but often follow learned predictions blindly, making performance unreliable when predictions are inaccurate. In contrast, emerging learning-augmented caching algorithms~\cite{pmlr-v80-lykouris18a,mitzenmacher2022algorithms} provide performance guarantees by carefully integrating predictions into caching policies, achieving both \emph{consistency} (near-optimality under perfect predictions) and \emph{robustness} (bounded worst-case performance under prediction errors). However, deployment remains challenging. A practical algorithm should satisfy strict time and space efficiency constraints, which some theoretical work overlooks, while also incurring low deployment overhead. We propose learning-augmented LRU, a deployment-oriented learning-augmented caching algorithm that guarantees \emph{1-consistency} and \emph{$O(k)$-robustness}, incurs low time and space overhead, and maintains strong compatibility. We further build a GPU cache, called \textsc{LCR}, on top of learning-augmented LRU to benefit from its theoretical guarantees and translate them into practical performance. In experiments, \textsc{LCR} reduces P99 time-to-first-token (TTFT) by up to 28.3\% on LLM workloads and increases throughput by up to 24.2\% on deep learning recommendation (DLRM) workloads. Even with poor predictions, performance degrades gracefully and remains close to \textsc{LRU}, demonstrating robustness with practical value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。