通过预计算用户行为前缀,显著提升长序列生成推荐的实时性能。
RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference
- 预缓存用户长期行为特征,避免每次推荐重复计算。
- 在固定延迟约束下,支持1.5倍更长的输入序列,吞吐提升3.6倍。
- 适合高并发、低延迟要求的工业级推荐系统场景。
实时推荐系统需在严格尾部延迟SLO内完成多阶段流程(检索、预处理、细粒度排序),排序阶段仅剩数十毫秒。生成式推荐(GR)模型可通过长用户行为序列提升质量,但生产环境中其序列长度受排序阶段P99延迟预算限制。我们发现,多数GR token编码的用户行为与候选物品无关,因此可提前计算并复用用户行为前缀。实现此方案面临挑战:前缀缓存需跨多个流水线阶段保留,用户规模导致缓存需求远超单设备容量,且盲目预推理会压垮共享资源。我们提出RelayGR,一种基于HBM的跨阶段接力推理系统。它选择性预推理长时用户前缀,将键值缓存保留在HBM中直至请求生命周期结束,确保后续排序无需远程加载。RelayGR结合三项技术:1)序列感知触发器,仅对潜在超时请求预推理,控制缓存与负载;2)亲和感知路由器,将预推理信号与排序请求路由至同一实例,实现缓存就近使用;3)内存感知扩展器,利用服务器本地DRAM捕获跨请求短期复用,避免冗余重载。我们在华为Ascend NPU上实现并评估该系统,在固定P99 SLO下,支持最长序列达1.5倍,合规吞吐最高提升3.6倍。
原文摘要 · Abstract (English)
Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for ranking. Generative recommendation (GR) models can improve quality by consuming long user-behavior sequences, but in production their online sequence length is tightly capped by the ranking-stage P99 budget. We observe that the majority of GR tokens encode user behaviors that are independent of the item candidates, suggesting an opportunity to pre-infer a user-behavior prefix once and reuse it during ranking rather than recomputing it on the critical path. Realizing this idea at industrial scale is non-trivial: the prefix cache must survive across multiple pipeline stages before the final ranking instance is determined, the user population implies cache footprints far beyond a single device, and indiscriminate pre-inference would overload shared resources under high QPS. We present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR selectively pre-infers long-term user prefixes, keeps their KV caches resident in HBM over the request lifecycle, and ensures the subsequent ranking can consume them without remote fetches. RelayGR combines three techniques: 1) a sequence-aware trigger that admits only at-risk requests under a bounded cache footprint and pre-inference load, 2) an affinity-aware router that co-locates cache production and consumption by routing both the auxiliary pre-infer signal and the ranking request to the same instance, and 3) a memory-aware expander that uses server-local DRAM to capture short-term cross-request reuse while avoiding redundant reloads. We implement RelayGR on Huawei Ascend NPUs and evaluate it with real queries. Under a fixed P99 SLO, RelayGR supports up to 1.5$\times$ longer sequences and improves SLO-compliant throughput by up to 3.6$\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。