提示词重写对密集检索效果影响因领域而异,可能提升或损害性能。
Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval
- 仅基于提示词的单步重写,不依赖检索反馈。
- 在FiQA上性能下降9.0%,TREC-COVID上提升5.1%,SciFact无显著变化。
- 效果取决于替换方向是否匹配领域术语,非替换行为本身。
仅通过提示词进行单步大模型查询重写(无需检索反馈)广泛应用于生产级RAG流程,但其对密集检索的影响尚不明确。本研究在三个BEIR基准、两种密集检索器及多种训练配置下开展系统性实证分析,发现效果具有强领域依赖性:在FiQA上nDCG@10下降9.0%,TREC-COVID上提升5.1%,SciFact则无显著变化。我们识别出一致机制:性能下降伴随重写后查询与相关文档间的词汇对齐度降低,因重写替换了已良好匹配查询中的领域特异性术语;性能提升则源于重写使查询更贴近语料库偏好术语并解决命名不一致问题。跨所有结果组,95%的重写发生词汇替换,表明有效性取决于替换方向而非替换本身。我们还研究了选择性重写,发现基于简单特征的门控虽能缓解最差情况退化,但无法稳定优于从不重写,即使使用理想选择器也仅带来小幅提升。总体而言,该结果表明,在优化良好的垂直领域中,提示词重写可能有害,建议在有标注或隐式反馈时采用领域自适应后训练策略。
原文摘要 · Abstract (English)
Prompt-only, single-step LLM query rewriting, where a rewrite is generated from the query alone without retrieval feedback, is commonly used in production RAG pipelines, but its effect on dense retrieval is poorly understood. We present a systematic empirical study across three BEIR benchmarks, two dense retrievers, and multiple training configurations, and find strongly domain-dependent behavior: rewriting degrades nDCG@10 by 9.0 percent on FiQA, improves it by 5.1 percent on TREC-COVID, and has no significant effect on SciFact. We identify a consistent mechanism: degradations co-occur with reduced lexical alignment between rewritten queries and relevant documents, as rewriting replaces domain-specific terms in already well-matched queries. In contrast, improvements arise when rewriting shifts queries toward corpus-preferred terminology and resolves inconsistent nomenclature. Lexical substitution occurs in 95 percent of rewrites across all outcome groups, showing that effectiveness depends on the direction of substitution rather than substitution itself. We also study selective rewriting and find that simple feature-based gating can reduce worst-case regressions but does not reliably outperform never rewriting, with even oracle selection offering only modest gains. Overall, these results show that prompt-only rewriting can be harmful in well-optimized verticals and suggest that domain-adaptive post-training is a safer strategy when supervision or implicit feedback is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。