arXiv:2603.02565cs.IRcs.CL2026-03

通过并行联合评估,显著降低大候选集生成的延迟。

FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation

  • 设计联合评估器,单次前向传播完成所有候选序列评分
  • 在K=50时推理延迟降44%,每秒请求数提升114%
  • 适合大规模推荐与文本生成场景,尤其有重复内容的场景

生成-评估(G-E)框架生成K个候选序列,由评估器选出得分最高的一个,广泛用于推荐系统和自然语言处理。现有评估器通常独立打分,虽可批量处理,但无法建模候选间交互,且重复计算请求级上下文与重复候选元素,导致总评估工作量随K近似线性增长。为此,我们提出FlashEvaluator,一种在单次前向传播中对所有候选序列进行联合评分的评估器。其将评估分解为共享请求级编码、可复用的候选侧计算、通过索引的序列组装以及跨序列交互以实现集合比较。该方案称为QKV-Cache:受自回归KV缓存启发,跨候选序列复用上下文侧的键值表示,当候选元素重复出现时,复用其条件化的查询侧表示。在重复项场景下,主要项编码成本取决于不同项的数量而非其总出现次数,从而降低新增候选的边际开销。我们进行了计算分析,并在推荐与文本摘要任务上评估了FlashEvaluator。结果表明,在保持推荐与摘要质量竞争力的同时,显著降低延迟并提高吞吐量。在快手线上部署中,当K=50时,相比生产基线,推理延迟降低44%,每秒请求数(QPS)提升114%,同时在留存率、用户活跃度及生态指标上取得统计显著提升。

原文摘要 · Abstract (English)

The Generator-Evaluator (G-E) framework generates K candidate sequences and uses an evaluator to select the highest-scoring one, which is widely used in recommender systems (RecSys) and natural language processing (NLP). Existing evaluators commonly score candidates independently. Although such evaluations can be batched, independent scoring neither models interactions among candidates nor eliminates repeated computation of request-level context and recurring candidate elements, causing the total evaluation work to grow approximately linearly with K. To handle with, we propose FlashEvaluator, a joint evaluator that scores all candidate sequences in a single forward pass. FlashEvaluator factorizes evaluation into shared request-level encoding, reusable candidate-side computation, sequence assembly by indexing, and cross-sequence interaction for setwise comparison. We call this request-local reuse scheme QKV-Cache: inspired by autoregressive KV caching, it reuses context-side key/value representations across candidate sequences and, when candidate elements recur, reuses their request-conditioned representations on the query side. In repeated-item settings, the dominant item-encoding cost therefore depends on the number of distinct items rather than their total occurrences across sequences, reducing the marginal cost of evaluating additional candidates. We provide a computational analysis and evaluate FlashEvaluator on recommendation and text summarization. The results show lower latency and higher throughput with competitive recommendation and summarization quality. In an online deployment at Kuaishou with K=50, FlashEvaluator reduces inference latency by 44% and increases QPS by 114% relative to the production baseline, while yielding statistically significant gains in retention, engagement, and ecosystem metrics.

推荐系统评估优化并行计算大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。