大规模下关键词检索比复杂搜索更高效,BM25成最优选择。
BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

- 控制语料规模变化28级,对比不同检索方法在相同条件下的表现。
- 1000万词以上时BM25准确率超其他方法近20点,且查询开销最低。
- 适合追求高可扩展性和低延迟的系统设计者,尤其大型知识库应用。
检索增强生成(RAG)包含词法、稠密检索、图索引和代理式搜索等多种范式,但通常在不同基准和单一语料规模下评估,导致其精度-成本的缩放特性不清晰。为此,我们进行了一项受控研究,将语料规模分为28个严格嵌套层级,覆盖约450倍的变化范围,同时保持问题和基础相关与对抗性文档不变。在固定阅读器模型和评判协议下,测量官方准确率、构建与查询的标记数及延迟。结果揭示出一种依赖规模的交叉点而非绝对优势:文件系统代理在最小共享层级领先,但在全规模下查询标记数高出39倍,且随搜索空间增大而效果下降。约1000万词时,BM25超越代理并持续领先,最大差距达20点。稠密检索虽高效但准确率较低;图式RAG在部署前即遇构建瓶颈,其可扩展变体在共享层级仍落后于BM25。总体而言,语料增长愈发倾向于全局候选排序:词法检索是最佳可扩展默认方案,而代理推理更适合在排序后发现阶段使用,而非替代排序。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search, but these paradigms are usually evaluated on different benchmarks at one corpus size, leaving their accuracy-cost scaling unclear. To bridge this gap, we present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold, while holding questions and a fixed bedrock of relevant and adversarial documents unchanged. Under one reader model and one judging protocol, we measure official accuracy, construction and query tokens, and latency. The results reveal a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers, but its sequential exploration costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale. BM25 also anchors the low-cost end of the Pareto frontier without LLM-based construction. Dense retrieval remains efficient but less accurate, whereas graph-based RAG encounters construction walls before deployment scale and its scalable variants remain below BM25 at shared tiers. Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。