arXiv:2603.10473cs.CLcs.AI2026-03KDD被引 1

用多维奖励机制让大模型更懂用户搜索需求,提升生成质量与体验。

Aligning Large Language Models with Searcher Preferences

论文配图:Aligning Large Language Models with Searcher Preferences
图 1 · 摘自论文原文
  • 分层设计奖励系统,区分基础约束与行为优化目标。
  • 上线后有效提升1.03%有效消费率,降低2.81%重搜率。
  • 适合关注生成式搜索、个性化推荐的工程师与研究者。

从以项目为中心的排序转向以答案为中心的合成,正在重塑搜索引擎的角色。尽管工业界已将生成技术应用于电商场景的封闭集项目排序,但在大型内容平台开展开放式生成式搜索的研究与部署仍有限。该场景面临噪声检索鲁棒性、不可妥协的安全保障及多样用户需求对齐等挑战。本文提出SearchLLM,首个面向开放式生成式搜索的大语言模型。设计分层多维奖励体系,将事实依据、基础回答质量与格式合规等底线约束,与抗噪声检索鲁棒性、用户需求对齐等行为优化目标分离。具体而言,奖励模型基于用户查询、会话历史和检索证据集评估回答,结合规则检查与人工校准的LLM判官,输出可解释的多维评分向量。引入门控聚合策略,结合组相对策略优化(GRPO)训练SearchLLM。在红笔记AI搜索入口部署后,离线评估与线上A/B测试显示生成质量与用户参与度显著提升,有效消费率提高1.03%,重搜率下降2.81%,同时严格保障安全与可靠性。

原文摘要 · Abstract (English)

The paradigm shift from item-centric ranking to answer-centric synthesis is redefining the role of search engines. While recent industrial progress has applied generative techniques to closed-set item ranking in e-commerce, research and deployment of open-ended generative search on large content platforms remain limited. This setting introduces challenges, including robustness to noisy retrieval, non-negotiable safety guarantees, and alignment with diverse user needs. In this work, we introduce SearchLLM, the first large language model (LLM) for open-ended generative search. We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints, including factual grounding, basic answer quality and format compliance, from behavior optimization objectives that promote robustness to noisy retrieval and alignment with user needs. Concretely, our reward model evaluates responses conditioned on the user query, session history, and retrieved evidence set, combining rule-based checks with human-calibrated LLM judges to produce an interpretable score vector over these dimensions. We introduce a Gated Aggregation Strategy to derive the training reward for optimizing SearchLLM with Group Relative Policy Optimization (GRPO). We deploy SearchLLM in the AI search entry of RedNote. Offline evaluations and online A/B tests show improved generation quality and user engagement, increasing Valid Consumption Rate by 1.03% and reducing Re-search Rate by 2.81%, while upholding strict safety and reliability standards.

生成式搜索大模型奖励机制用户对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。