用统一语义建模替代传统关键词索引，提升搜索准确性
UniDex: Rethinking Search Inverted Indexing with Unified Semantic Modeling
- 用语义编码代替关键词匹配，实现更深层的语义检索
- 在真实流量中显著提升检索效果，支持亿级用户并发
- 适合需要高精度语义搜索的推荐与内容平台
传统倒排索引依赖精确词项匹配，易受表面词重叠影响，限制了泛化能力与检索效果。为此，我们提出UniDex，一种基于模型的新型倒排索引方法，通过统一语义建模实现范式革新。其核心由两部分构成：UniTouch将查询与文档映射为语义ID以增强检索，UniRank利用语义匹配实现高效排序。在大规模工业数据集及真实线上流量评估中，UniDex显著提升检索性能，标志着从词项驱动向模型驱动索引的转变。该系统已部署于快手短视频搜索场景，稳定服务数亿活跃用户。
原文摘要 · Abstract (English)
Inverted indexing has traditionally been a cornerstone of modern search systems, leveraging exact term matches to determine relevance between queries and documents. However, this term-based approach often emphasizes surface-level token overlap, limiting the system's generalization capabilities and retrieval effectiveness. To address these challenges, we propose UniDex, a novel model-based method that employs unified semantic modeling to revolutionize inverted indexing. UniDex replaces complex manual designs with a streamlined architecture, enhancing semantic generalization while reducing maintenance overhead. Our approach involves two key components: UniTouch, which maps queries and documents into semantic IDs for improved retrieval, and UniRank, which employs semantic matching to rank results effectively. Through large-scale industrial datasets and real-world online traffic assessments, we demonstrate that UniDex significantly improves retrieval capabilities, marking a paradigm shift from term-based to model-based indexing. Our deployment within Kuaishou's short-video search systems further validates UniDex's practical effectiveness, serving hundreds of millions of active users efficiently.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。