十种大模型查询重写方法在统一框架下对比，发现效果依赖检索方式。
A Reproducibility Study of LLM-Based Query Reformulation
- 统一实验框架对比十种重写方法，控制模型规模与检索范式。
- 词法检索中有效的方法在神经检索中不一致，大模型未必更优。
- 开源所有配置与评测脚本，支持透明复现与持续比较。
大语言模型（LLM）广泛用于信息检索中的查询重写与扩展，许多研究声称取得了显著性能提升。然而这些结果多在异构实验条件下获得，难以判断其可复现性或是否依赖特定实现。本文在统一、严格控制的实验框架下，系统性地复现并对比了十种代表性LLM查询重写方法。评估涵盖两类模型架构、两个参数规模、三种检索范式（词法、学习稀疏、稠密）及九个基准数据集，覆盖TREC Deep Learning和BEIR。结果表明，重写增益高度依赖于检索范式；词法检索中有效的策略在神经检索中并不一致；更大规模的模型也未带来统一性能提升。研究揭示了以往成果的稳定性与局限性。为促进透明复现与持续比较，我们通过QueryGym开源平台发布所有提示、配置、评估脚本与运行文件，并提供公开排行榜。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are now widely used for query reformulation and expansion in Information Retrieval, with many studies reporting substantial effectiveness gains. However, these results are typically obtained under heterogeneous experimental conditions, making it difficult to assess which findings are reproducible and which depend on specific implementation choices. In this work, we present a systematic reproducibility and comparative study of ten representative LLM-based query reformulation methods under a unified and strictly controlled experimental framework. We evaluate methods across two architectural LLM families at two parameter scales, three retrieval paradigms (lexical, learned sparse, and dense), and nine benchmark datasets spanning TREC Deep Learning and BEIR. Our results show that reformulation gains are strongly conditioned on the retrieval paradigm, that improvements observed under lexical retrieval do not consistently transfer to neural retrievers, and that larger LLMs do not uniformly yield better downstream performance. These findings clarify the stability and limits of reported gains in prior work. To enable transparent replication and ongoing comparison, we release all prompts, configurations, evaluation scripts, and run files through QueryGym, an open-source reformulation toolkit with a public leaderboard.\footnote{https://leaderboard.querygym.com}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。