arXiv:2511.13885cs.IR2025-11被引 2

用强化学习动态优化电商搜索的语义检索,不依赖复杂负样本构造。

Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Search

  • 训练时动态召回候选商品,替代传统离线负样本构建
  • 结合LLM评分与商品质量等多目标奖励,提升长尾查询效果
  • 适合追求高效迭代与真实用户偏好对齐的工业级搜索系统

稠密检索作为电商搜索引擎的核心组件,通过预训练嵌入模型将用户查询与商品映射到统一语义空间,实现大规模实时语义检索。尽管大语言模型(LLM)正逐步取代传统BERT架构进行嵌入表示,其训练范式仍沿用BERT式的监督微调和硬负例挖掘策略。该方法依赖复杂的离线硬负样本构建流程,制约了模型迭代效率,限制了语义表示能力的演化潜力。此外,现有多任务学习框架在同时优化语义相关性与非相关性目标时面临此消彼长效应。本文提出Retrieval-GRPO,一种基于多目标强化学习的稠密检索框架,以解决上述挑战。该方法通过在训练过程中动态为每个查询检索Top-K候选商品,消除对离线硬负样本的依赖,并引入一个相关性LLM作为奖励模型,生成实时反馈信号。具体而言,检索模型通过强化学习动态优化嵌入表示,奖励信号融合LLM生成的相关性评分、商品质量评分及多向排他性指标,实现多目标用户偏好对齐与实时错误修正。该机制不仅摆脱了对硬负例的依赖,还通过协同多目标优化缓解了此消彼长问题,显著提升了模型对复杂长尾查询的语义泛化能力。大量离线与在线实验验证了Retrieval-GRPO的有效性,该模型已部署于中国最大电商平台。

原文摘要 · Abstract (English)

Dense retrieval, as the core component of e-commerce search engines, maps user queries and items into a unified semantic space through pre-trained embedding models to enable large-scale real-time semantic retrieval. Despite the rapid advancement of LLMs gradually replacing traditional BERT architectures for embedding, their training paradigms still adhere to BERT-like supervised fine-tuning and hard negative mining strategies. This approach relies on complex offline hard negative sample construction pipelines, which constrain model iteration efficiency and hinder the evolutionary potential of semantic representation capabilities. Besides, existing multi-task learning frameworks face the seesaw effect when simultaneously optimizing semantic relevance and non-relevance objectives. In this paper, we propose Retrieval-GRPO, a multi-objective reinforcement learning-based dense retrieval framework designed to address these challenges. The method eliminates offline hard negative sample construction by dynamically retrieving Top-K candidate products for each query during training, while introducing a relevance LLM as a reward model to generate real-time feedback. Specifically, the retrieval model dynamically optimizes embedding representations through reinforcement learning, with reward signals combining LLM-generated relevance scores, product quality scores, and multi-way exclusivity metrics to achieve multi-objective user preference alignment and real-time error correction. This mechanism not only removes dependency on hard negatives but also mitigates the seesaw effect through collaborative multi-objective optimization, significantly enhancing the model's semantic generalization capability for complex long-tail queries. Extensive offline and online experiments validate the effectiveness of Retrieval-GRPO, which has been deployed on China's largest e-commerce platform.

稠密检索强化学习电商搜索多目标优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。