arXiv:2602.22647cs.IRcs.CL2026-02KDD被引 10

用稀疏矩阵加速前缀树,让大模型推荐更高效且严格受限。

Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

  • 将前缀树转为稀疏矩阵,实现硬件加速下的向量化解码
  • 在工业级视频推荐系统中延迟仅0.033毫秒/步,提速达948倍
  • 适合需要严格约束输出的高并发生成式推荐场景

生成式检索已成为基于大模型推荐的有力范式。然而,工业推荐系统常需根据业务逻辑(如强制内容新鲜度或品类限制)对输出空间进行约束,标准自回归解码无法原生支持。现有基于前缀树(Trie)的约束解码方法在硬件加速器(TPU/GPU)上带来严重延迟。本文提出STATIC(稀疏转移矩阵加速的前缀树索引),一种专为TPU/GPU上的高吞吐生成式检索设计的高效可扩展约束解码技术。通过将前缀树扁平化为静态压缩稀疏行(CSR)矩阵,将不规则的树遍历转化为完全向量化的稀疏矩阵运算,显著提升硬件加速效率。我们在服务数十亿用户的大型工业视频推荐平台部署了STATIC,实现了显著的产品指标提升,每步延迟仅0.033毫秒,占推理时间0.25%,相比CPU Trie实现948倍加速,比硬件加速的二分查找基线快47至1033倍。此外,STATIC在多种实际配置下运行开销极低。据我们所知,STATIC首次实现了严格约束生成式检索的生产级部署。学术基准测试也表明,STATIC能显著改善生成式检索的冷启动表现。代码已开源:https://github.com/youtube/static-constraint-decoding。

原文摘要 · Abstract (English)

Generative retrieval has emerged as a powerful paradigm for LLM-based recommendation. However, industrial recommender systems often benefit from restricting the output space to a constrained subset of items based on business logic (e.g. enforcing content freshness or product category), which standard autoregressive decoding cannot natively support. Moreover, existing constrained decoding methods that make use of prefix trees (Tries) incur severe latency penalties on hardware accelerators (TPUs/GPUs). In this work, we introduce STATIC (Sparse Transition Matrix-Accelerated Trie Index for Constrained Decoding), an efficient and scalable constrained decoding technique designed specifically for high-throughput LLM-based generative retrieval on TPUs/GPUs. By flattening the prefix tree into a static Compressed Sparse Row (CSR) matrix, we transform irregular tree traversals into fully vectorized sparse matrix operations, unlocking massive efficiency gains on hardware accelerators. We deploy STATIC on a large-scale industrial video recommendation platform serving billions of users. STATIC produces significant product metric impact with minimal latency overhead (0.033 ms per step and 0.25% of inference time), achieving a 948x speedup over a CPU trie implementation and a 47-1033x speedup over a hardware-accelerated binary-search baseline. Furthermore, the runtime overhead of STATIC remains extremely low across a wide range of practical configurations. To the best of our knowledge, STATIC enables the first production-scale deployment of strictly constrained generative retrieval. In addition, evaluation on academic benchmarks demonstrates that STATIC can considerably improve cold-start performance for generative retrieval. Our code is available at https://github.com/youtube/static-constraint-decoding.

生成式推荐约束解码稀疏矩阵加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。