arXiv:2608.19595cs.IR2026-08

用语义标识增强强化学习,提升电商稠密检索的精准度

SSR-GRPO: Integrating Supervision and Semantic IDs into Reinforcement Learning for Dense Retrieval in E-commerce

论文配图:SSR-GRPO: Integrating Supervision and Semantic IDs into Reinforcement Learning for Dense Retrieval in E-commerce
图 1 · 摘自论文原文
  • 引入语义标识与双重评估机制,减少奖励模型偏差
  • 通过硬负样本挖掘和掩码函数,过滤噪声候选结果
  • 适合追求高精度检索的电商搜索系统开发者

基于嵌入的检索(EBR)在电商搜索中至关重要,但常难以处理复杂语义。现有方法虽微调大语言模型(LLMs)进行表征学习,却缺乏应对复杂隐含语义的稳健机制。尽管最近的Retrieval-GRPO将强化学习引入稠密检索,但仍受限于批量采样不足导致的噪声Top-K候选,以及使用同训练的LLM作为奖励模型引发的相关性评估偏差。为此,我们提出融合监督信号与语义标识的SSR-GRPO方法。首先,设计双视角相关性评估框架,结合量化学习生成的语义标识(SIDs)与稠密向量,生成更无偏的相关性评分。其次,利用生成的SIDs的层级相似关系,挖掘一组难负样本:其一用于构建集成于R-GRPO的掩码函数,有效过滤组内噪声样本;其二用于构建包含正负样本对的Retrieval-DPO任务,使模型从成对视角捕捉细粒度语义差异。通过整合这些优化策略,提出SSR-GRPO。大量离线与在线实验验证其有效性,并已在大规模电商平台上部署。

原文摘要 · Abstract (English)

Embedding-based retrieval (EBR) is pivotal in e-commerce search but often struggles with complex semantics. While recent methods often fine-tune large language models (LLMs) for representation learning, they typically lack robust mechanisms for handling complex and implicit semantics. While Retrieval-GRPO (R-GRPO) recently introduced reinforcement learning to dense retrieval, it suffers from noisy top-K candidates due to limited batch sampling and biased relevance assessments caused by using similarly trained LLMs as reward models. To tackle these issues, we propose Supervised Retrieval-GRPO with Semantic Identifiers (SSR-GRPO). Specifically, our method first proposes a dual-perspective framework for relevance assessment. It leverages both Semantic Identifiers (SIDs) produced by quantization learning and dense representation vectors to generate more unbiased relevance scores. Furthermore, leveraging the hierarchical similarity relationships of the generated SIDs, we mine a set of hard negative samples that serve two purposes: (1) to design a masking function integrated into R-GRPO, effectively filtering intra-group noisy samples; and (2) to construct a Retrieval-DPO task composed of positive and negative sample pairs, enabling the model to capture fine-grained semantic distinctions from a pair-wise perspective. By integrating these optimization strategies, we propose SSR-GRPO. Extensive offline and online experiments validate SSR-GRPO's effectiveness, and it has been deployed on a large-scale e-commerce platform.

稠密检索强化学习电商搜索语义标识

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。