arXiv:2512.00968cs.IRcs.AI2025-12KDD被引 13

用强化学习让搜索模型像人一样推理,提升推荐准确性。

Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search

  • 引入强化学习优化生成式排序模型的推理过程。
  • 在小红书搜索中实现关键指标显著提升,线上测试效果稳定。
  • 适合关注搜索系统优化与可解释性建模的工业界研究者。

排序相关性是搜索引擎的核心任务,旨在识别与用户查询最相关的项目。传统相关性模型通常生成标量分数或直接预测相关标签,限制了可解释性及复杂相关信号的建模能力。受链式思维(CoT)推理在复杂任务中进展的启发,我们探究显式推理是否能提升相关性建模的可解释性与性能。然而,现有基于推理的生成式相关性模型(GRMs)主要依赖大规模人工标注或合成的CoT数据进行监督微调,导致泛化能力有限。此外,通用、自由形式的推理往往过于笼统且缺乏实际依据,难以应对开放域搜索中普遍存在的多样化和模糊场景。本文将小红书搜索中的相关性建模视为推理任务,提出一种基于强化学习(RL)的训练框架,以增强GRMs的接地推理能力。具体而言,我们将实际业务相关的相关性标准融入多步推理提示设计,并提出轻量级的逐步优势掩码(SAM)策略,通过改进信用分配促进对这些标准的有效学习。为支持工业部署,我们进一步将大规模RL调优模型压缩为轻量版本,适用于真实搜索系统。大量离线评估与在线A/B测试表明,该方法在关键相关性和业务指标上持续取得显著提升,验证了其在大规模工业搜索系统中的有效性、鲁棒性与实用性。

原文摘要 · Abstract (English)

Ranking relevance is a fundamental task in search engines, aiming to identify the items most relevant to a given user query. Traditional relevance models typically produce scalar scores or directly predict relevance labels, limiting both interpretability and the modeling of complex relevance signals. Inspired by recent advances in Chain-of-Thought (CoT) reasoning for complex tasks, we investigate whether explicit reasoning can enhance both interpretability and performance in relevance modeling. However, existing reasoning-based Generative Relevance Models (GRMs) primarily rely on supervised fine-tuning on large amounts of human-annotated or synthetic CoT data, which often leads to limited generalization. Moreover, domain-agnostic, free-form reasoning tends to be overly generic and insufficiently grounded, limiting its potential to handle the diverse and ambiguous cases prevalent in open-domain search. In this work, we formulate relevance modeling in Xiaohongshu search as a reasoning task and introduce a Reinforcement Learning (RL)-based training framework to enhance the grounded reasoning capabilities of GRMs. Specifically, we incorporate practical business-specific relevance criteria into the multi-step reasoning prompt design and propose Stepwise Advantage Masking (SAM), a lightweight process-supervision strategy which facilitates effective learning of these criteria through improved credit assignment. To enable industrial deployment, we further distill the large-scale RL-tuned model to a lightweight version suitable for real-world search systems. Extensive offline evaluations and online A/B tests demonstrate that our approach consistently delivers significant improvements across key relevance and business metrics, validating its effectiveness, robustness, and practicality for large-scale industrial search systems.

搜索排序强化学习生成模型可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。