提出可复现的推理型检索基准，发现查询加权显著提升长查询效果。
Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM
- 引入查询侧BM25（BM25Q）提升长文本检索性能
- 在BRIGHT上验证了新基线优于传统方法，准确率提升显著
- 适配多个开源工具链，便于研究者复现与对比
针对大语言模型的检索任务应反映推理密集型的长查询特点，本文系统研究了以推理为导向的BRIGHT检索基准，并将其强而可复现的基准方法集成至Anserini、Pyserini和RankLLM中。评估了词法、稀疏、稠密及融合型检索器，以及大语言模型重排序器在长查询场景下的表现。在复现BRIGHT的词法基线时，发现一个关键但未充分记录的细节：查询侧BM25（BM25Q），即对查询本身应用BM25加权。在多句长查询下，BM25Q持续优于标准BM25，成为推理导向检索中最优的词法基线。进一步审计BRIGHT语料库，发现影响评估的数据质量问题并提出缓解方案。最后，在五个额外基准上测试BM25Q的泛化能力，发现其增益主要局限于BRIGHT，而与标准BM25融合则在多数数据集上表现最稳定。
原文摘要 · Abstract (English)
Retrieval benchmarks for large language models (LLMs) should reflect the long, reasoning-intensive queries typical of retrieval-augmented generation (RAG). We present a systematic study of BRIGHT, a reasoning-focused retrieval benchmark, along with strong, reproducible reference methods integrated into Anserini, Pyserini, and RankLLM. We evaluate lexical, sparse, dense, and fusion-based retrievers, as well as LLM rerankers, under long-query settings. In reproducing BRIGHT's lexical baseline, we identify a key under-documented detail: query-side BM25 (BM25Q), which applies BM25 weighting to the query itself. On long, multi-sentence queries, BM25Q consistently outperforms standard BM25, making it the strongest lexical baseline for reasoning-oriented retrieval. We further audit the BRIGHT corpus, uncovering data quality issues that impact evaluation, and offer mitigation. Finally, we study the generalizability of BM25Q across five additional benchmarks, finding its gains largely specific to BRIGHT, while fusion with standard BM25 provides the most consistent improvements across datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。