用缓存机制优化GPU嵌入存储，提升大规模推荐系统效率
HierarchicalKV: A GPU Hash Table with Cache Semantics for Continuous Online Embedding Storage
- 采用缓存语义替代传统字典模式，支持按策略淘汰旧数据
- 单卡每秒处理39亿次查找，负载率0.5至1.0时吞吐稳定
- 适合大规模在线推荐系统，已集成至多个开源框架
传统GPU哈希表会保留所有插入的键，导致在嵌入表超出单卡显存容量时浪费宝贵的高带宽内存（HBM）。本文挑战这一假设，引入缓存语义，将策略驱动的淘汰作为核心操作。提出HierarchicalKV（HKV），首个通用的、以缓存语义为常态的GPU哈希表库：每次满桶插入（更新或插入）均通过淘汰或拒绝接纳就地解决，而非重新散列或因容量不足失败。HKV协同设计四种核心机制——缓存行对齐的桶、内联评分驱动插入、基于评分的动态双桶选择、三组并发控制，并采用分层键值分离以突破HBM限制。在NVIDIA H100 NVL GPU上，HKV实现最高39亿键值对/秒（B-KV/s）的查找吞吐，负载因子0.50–1.00间波动小于5%，在λ=0.50时比WarpCore（最强的字典语义基线）快1.4倍，较间接寻址基线快2.6至9.4倍。自2022年10月开源以来，已集成至多个开源推荐框架。
原文摘要 · Abstract (English)
Traditional GPU hash tables preserve every inserted key -- a dictionary assumption that wastes scarce High Bandwidth Memory (HBM) when embedding tables routinely exceed single-GPU capacity. We challenge this assumption with cache semantics, where policy-driven eviction is a first-class operation. We introduce HierarchicalKV (HKV), the first general-purpose GPU hash table library whose normal full-capacity operating contract is cache-semantic: each full-bucket upsert (update-or-insert) is resolved in place by eviction or admission rejection rather than by rehashing or capacity-induced failure. HKV co-designs four core mechanisms -- cache-line-aligned buckets, in-line score-driven upsert, score-based dynamic dual-bucket selection, and triple-group concurrency -- and uses tiered key-value separation as a scaling enabler beyond HBM. On an NVIDIA H100 NVL GPU, HKV achieves up to 3.9 billion key-value pairs per second (B-KV/s) find throughput, stable across load factors 0.50-1.00 (<5% variation), and delivers 1.4x higher find throughput than WarpCore (the strongest dictionary-semantic GPU baseline at lambda=0.50) and up to 2.6-9.4x over indirection-based GPU baselines. Since its open-source release in October 2022, HKV has been integrated into multiple open-source recommendation frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。