提出可实时更新的多面嵌入索引,彻底消除推荐系统中的近似最近邻搜索
Rethinking ANN-based Retrieval: Multifaceted Learnable Index for Large-scale Recommendation System
- 在统一框架中联合学习物品嵌入与分层量化索引,实现端到端优化
- 线上实验显示召回率提升最高11.8%,冷启动内容曝光量增57.29%
- 适用于高并发、需快速响应的工业级推荐系统,尤其适合新内容冷启动场景
近似最近邻(ANN)搜索广泛用于大规模推荐系统的检索阶段。该阶段通过学习得到的物品嵌入向量对候选物品进行索引,再对每个用户(或物品)查询执行ANN搜索以获取相关物品集合。然而,基于ANN的检索存在两大局限:一是物品嵌入与索引通常分阶段训练,索引常在嵌入训练完成后离线构建,导致检索质量不佳,尤其对新生成物品表现差;二是尽管ANN可实现亚线性查询时间,仍需为每次请求执行,大规模部署下计算开销巨大。本文提出多面可学习索引(MFLI),一种可扩展、实时的检索范式,能够在统一框架中联合学习多面物品嵌入与索引,并在推理时完全消除ANN搜索。具体地,通过残差量化构建分层代码本,并与嵌入联合训练。同时引入高效的多面索引结构与支持实时更新的机制。推理时直接使用学习到的分层索引定位相关物品,无需运行ANN搜索。在包含数十亿用户的实际数据集上进行的大量实验表明,与现有最优方法相比,MFLI在参与度任务上召回率最高提升11.8%,冷内容分发效率最高提升57.29%,语义相关性提升13.5%。我们还在线上系统中部署MFLI,结果表明其显著提升用户参与度,降低流行度偏差,且服务效率更高。
原文摘要 · Abstract (English)
Approximate nearest neighbor (ANN) search is widely used in the retrieval stage of large-scale recommendation systems. In this stage, candidate items are indexed using their learned embedding vectors, and ANN search is executed for each user (or item) query to retrieve a set of relevant items. However, ANN-based retrieval has two key limitations. First, item embeddings and their indices are typically learned in separate stages: indexing is often performed offline after embeddings are trained, which can yield suboptimal retrieval quality-especially for newly created items. Second, although ANN offers sublinear query time, it must still be run for every request, incurring substantial computation cost at industry scale. In this paper, we propose MultiFaceted Learnable Index (MFLI), a scalable, real-time retrieval paradigm that learns multifaceted item embeddings and indices within a unified framework and eliminates ANN search at serving time. Specifically, we construct a multifaceted hierarchical codebook via residual quantization of item embeddings and co-train the codebook with the embeddings. We further introduce an efficient multifaceted indexing structure and mechanisms that support real-time updates. At serving time, the learned hierarchical indices are used directly to identify relevant items, avoiding ANN search altogether. Extensive experiments on real-world data with billions of users show that MFLI improves recall on engagement tasks by up to 11.8\%, cold-content delivery by up to 57.29\%, and semantic relevance by 13.5\% compared with prior state-of-the-art methods. We also deploy MFLI in the system and report online experimental results demonstrating improved engagement, less popularity bias, and higher serving efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。