提出可规模化高效运行的分层检索方法,兼顾透明性与性能。
Hierarchical Retrieval at Scale: Bridging Transparency and Efficiency
- 构建树状结构直接优化检索性能,支持多粒度语义分组。
- 在低延迟下实现比现有分层方法更高的检索准确率。
- 可灵活选择不同层级表示,适合对效率与可解释性有要求的场景。
信息检索是许多智能系统的核心,使输出能基于大规模新数据集进行条件生成。尽管有效,将数据编码为高维向量进行相似性搜索会带来巨大的内存和计算开销,并难以解析系统内部机制。分层检索通过多粒度组织数据提供了可解释性替代方案,但其效率和性能不及平坦检索方法。本文提出Retreever,一种基于树结构的分层检索方法,通过直接优化结构以提升检索性能,同时利用有意义的语义分组自然实现透明性。该方法可通过任意树层级索引数据,灵活权衡成本与效用。实验表明,Retreever在粗粒度(中间层级)和细粒度(终端层级)表示上均表现优异,且在所有分层方法中达到最高检索准确率和最低延迟,证明该技术在实际应用中的可行性。
原文摘要 · Abstract (English)
Information retrieval is a core component of many intelligent systems as it enables conditioning of outputs on new and large-scale datasets. While effective, the standard practice of encoding data into high-dimensional representations for similarity search entails large memory and compute footprints, and also makes it hard to inspect the inner workings of the system. Hierarchical retrieval methods offer an interpretable alternative by organizing data at multiple granular levels, yet do not match the efficiency and performance of flat retrieval approaches. In this paper, we propose Retreever, a tree-based method that makes hierarchical retrieval viable at scale by directly optimizing its structure for retrieval performance while naturally providing transparency through meaningful semantic groupings. Our method offers the flexibility to balance cost and utility by indexing data using representations from any tree level. We show that Retreever delivers strong coarse (intermediate levels) and fine representations (terminal level), while achieving the highest retrieval accuracy at the lowest latency among hierarchical methods. These results demonstrate that this family of techniques is viable in practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。