arXiv:2606.14127cs.IRcs.CL2026-06

持续微调大模型重写搜索词,提升视频搜索相关性。

CoRe: A Continuously Reward-Finetuned LLM Query Rewriter for Multi-Stage Context-Aware Relevance in Web-Scale Video Search

  • 用生产级融合算法设计奖励函数,贴近真实排序效果。
  • 每周处理数百万条数据,重写后查询失败率下降显著。
  • 适合需要持续优化搜索体验的工业级推荐系统团队。

基于大模型的搜索词重写系统在生产中面临矛盾:训练奖励需反映重写结果在真实排序器中的表现,但训练成本必须足够低以支持频繁更新。我们提出 CoRe(Context Relevance)系统,在主流短视频搜索引擎中连续部署超过五个月,每周更新一次。其奖励函数以部署的多模态相关性模型为依据,采用乘法比例形式模拟生产端融合逻辑,缩小了离线奖励代理与实际生产间的差距。通过半在线的混合偏好优化循环,使该奖励在每周百万级样本规模下仍具成本效益:采用类似 DPO 的成对目标,仅对采样轨迹中的 top-k/bottom-k 子集进行梯度传播,并通过阶段结构将训练器与推理服务器参数同步频率从每步降低至每阶段。一套自动化上线门控机制基于奖励与稳定性指标,成功检测并恢复了一次真实的奖励作弊事件。重写输出作为并行相关性信号应用于召回、rawrank 和 finerank 阶段,不取代原有信号,从而限制重写失效的影响范围。两次连续的线上 A/B 测试显示,先在 finerank 部署,再扩展至 recall 与 rawrank 后,受影响查询的变更查询率显著降低,所有核心相关性与用户参与度指标均朝预期方向改善。

原文摘要 · Abstract (English)

LLM-based query rewriters in production face a tension: the training reward must reflect how the rewrite is consumed by the production ranker, yet the training procedure must be cheap enough to support continuous redeployment as data drifts. We present CoRe (Context Relevance), such a system, redeployed weekly for over five months in a major short-video search engine. Our reward uses the deployed multimodal relevance model as its source and a multiplicative ratio form mirroring the production fusion algebra, closing the simulation-production gap that offline reward proxies leave open. A semi-online Mixed Preference Optimization loop makes this reward affordable at multi-million-instance weekly scale: a DPO-style pairwise objective restricts the gradient pass to a small top-k/bottom-k subset of sampled trajectories, and a phase structure reduces trainer/inference-server parameter syncs from per-step to per-phase. An automated promotion gate over reward-like and stability metrics detected and recovered from a real reward-hacking incident in production. Rewriter output is consumed as parallel relevance signals at recall, rawrank, and finerank without displacing the original signals, bounding rewriter-failure blast radius. Online A/B from two sequential production launches, first deploying the rewriter at finerank, then extending consumption to recall and rawrank, delivers statistically significant reductions in change-query rate on rewrite-impacted queries, with all headline relevance and engagement metrics moving in the expected direction.

搜索优化大模型应用在线学习推荐系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。