用策略引导强化学习,让查询重写更高效准确
SAGE: Strategy-Adaptive Generation Engine for Query Rewriting
- 以专家策略引导LLM生成,提升重写质量
- 在多个基准上达到SOTA的NDCG@10表现
- 自动选择最优策略,降低推理开销
查询重写对密集检索至关重要,但现有方法依赖大规模标注数据或效率低下的强化学习探索。本文发现,用语义扩展、实体消歧等少量专家策略引导大语言模型,可显著提升在HotpotQA、FEVER、NFCorpus和SciFact等挑战性数据集上的检索效果。基于此,提出策略自适应生成引擎SAGE,将其融入强化学习框架,并设计两种新奖励塑造机制:战略信用分配(SCS)与对比奖励塑造(CRS),提供更精准的学习信号。该方法不仅在NDCG@10上取得新SOTA,还涌现出关键能力:智能选择最优策略,减少无效探索,生成简洁重写,降低推理成本而不损失性能。结果表明,结合精细奖励塑造的策略引导强化学习,为构建下一代稳健信息检索系统提供了可扩展、高效且可解释的新范式。
原文摘要 · Abstract (English)
Query rewriting is pivotal for enhancing dense retrieval, yet current methods demand large-scale supervised data or suffer from inefficient reinforcement learning (RL) exploration. In this work, we first establish that guiding Large Language Models (LLMs) with a concise set of expert-crafted strategies, such as semantic expansion and entity disambiguation, substantially improves retrieval effectiveness on challenging benchmarks, including HotpotQA, FEVER, NFCorpus, and SciFact. Building on this insight, we introduce the Strategy-Adaptive Generation Engine (SAGE), which operationalizes these strategies in an RL framework. SAGE introduces two novel reward shaping mechanisms-Strategic Credit Shaping (SCS) and Contrastive Reward Shaping (CRS)-to deliver more informative learning signals. This strategy-guided approach not only achieves new state-of-the-art NDCG@10 results, but also uncovers a compelling emergent behavior: the agent learns to select optimal strategies, reduces unnecessary exploration, and generates concise rewrites, lowering inference cost without sacrificing performance. Our findings demonstrate that strategy-guided RL, enhanced with nuanced reward shaping, offers a scalable, efficient, and more interpretable paradigm for developing the next generation of robust information retrieval systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。