用图结构精准分配生成式搜索中的推理步骤功劳,提升电商推荐相关性。
Graph-GRPO: Dependency-Aware Credit Assignment for Generative E-commerce Search Relevance

- 构建推理依赖图,将思维链步骤建模为节点与边,捕捉逻辑关联。
- 通过图传播机制实现细粒度奖励分配,错误步骤定位更准确。
- 适合需要高精度推理的电商搜索系统,尤其擅长复杂匹配场景。
电商搜索的相关性建模需综合理解用户查询、商品信息及细粒度属性匹配。当前基于大语言模型的链式思维(CoT)方法常采用强化学习优化,但现有方法多依赖结果级奖励,将整个推理链视为单一优化单元,难以区分错误与正确中间步骤,导致信用分配失准。虽有过程奖励方法提供更密集监督,但通常忽略推理步骤间的依赖关系与错误传播,责任归属困难。本文提出Graph-GRPO,一种面向多组件相关性推理的图结构增强型GRPO算法。该方法构建相关性推理依赖图,将CoT步骤作为节点,逻辑依赖作为边,通过图结构传播最终奖励以生成步骤级信用信号,实现更精确的细粒度信用分配。引入主损失驱动控制器,自适应调节边级信用传播系数,并结合CoT随机掩码进行监督策略初始化及基于图节点的多头蒸馏,形成可训练且可部署的生成式相关性建模框架。在主流电商平台的离线评估与在线A/B测试中,该框架显著提升相关性分类指标与关键用户参与度指标。
原文摘要 · Abstract (English)
Search relevance modeling is a core task in e-commerce search systems, assessing how well a user query matches candidate products. Rather than relying on a single holistic matching signal, relevance judgment often requires structured reasoning over query understanding, product understanding, and facet-level matching. With large language models (LLMs), this process is increasingly formulated as chain-of-thought (CoT) reasoning and optimized with reinforcement learning (RL). However, existing RL methods mainly rely on outcome-level rewards and treat the entire reasoning chain as a single optimization unit. This makes it difficult to distinguish faulty reasoning steps from correct intermediate ones, leading to misaligned credit assignment. Although process-reward methods provide denser supervision, they often treat reasoning steps independently and ignore dependency-driven error propagation, making responsibility attribution difficult and limiting the optimization of structured relevance reasoning. We propose Graph-GRPO, a graph-structured extension of GRPO for multi-component relevance reasoning. Graph-GRPO constructs a relevance reasoning dependency graph, where CoT steps are modeled as nodes and their logical dependencies as edges. It propagates outcome-level rewards over the graph to derive step-level credit signals, enabling more accurate fine-grained credit assignment. We further introduce a main-loss-driven controller that adaptively adjusts edge-wise credit-propagation coefficients. Together with CoT random masking for supervised policy initialization and graph-node-based multi-head distillation, we build a trainable and deployable framework for generative relevance modeling. Extensive offline evaluations and online A/B tests on a leading e-commerce platform demonstrate that the Graph-GRPO-based framework improves relevance classification metrics and key engagement metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。