让大模型先给答案再解释,兼顾搜索精度与响应速度。
Answer First, Reason Later: Aligning Search Relevance via Mode-Balanced Reinforcement Learning
- 先输出答案再补逻辑解释,实现快速响应与可解释性平衡。
- 通过模式均衡优化,避免强化学习导致的规则遗忘问题。
- 可将专家模型知识高效压缩到小模型,适合线上部署。
构建低延迟高精度的搜索相关性模型是搜索领域的长期挑战。为满足在线系统毫秒级响应需求的同时保留大语言模型的可解释推理过程,本文提出「答案先行、推理后置」(AFRL)新范式:模型首字即输出明确的相关性评分,随后生成结构化逻辑说明。受推理模型成功启发,采用「监督微调(SFT)+ 强化学习(RL)」流程实现该范式。然而直接应用现有强化学习易引发模式坍缩,使模型忽略复杂长尾规则。从信息论视角看,强化学习最小化反向KL散度,倾向于寻找概率峰值(模式聚焦),易产生奖励欺骗;而监督微调最小化正向KL散度,能覆盖数据分布(模式覆盖),有效锚定专家规则。基于此,本文提出模式平衡优化策略,在分步梯度策略优化(Stepwise-GRPO)中引入SFT辅助损失以平衡两者特性。此外,构建自动化指令演化系统与多阶段课程训练机制,保障数据质量。大量实验表明,32B教师模型达到当前最优性能;且AFRL架构支持高效知识蒸馏,成功将专家级逻辑迁移至0.6B小型模型,实现推理深度与部署延迟的统一。
原文摘要 · Abstract (English)
Building a search relevance model that achieves both low latency and high performance is a long-standing challenge in the search industry. To satisfy the millisecond-level response requirements of online systems while retaining the interpretable reasoning traces of Large Language Models (LLMs), we propose a novel \textbf{Answer-First, Reason Later (AFRL)} paradigm. This paradigm requires the model to output the definitive relevance score in the very first token, followed by a structured logical explanation. Inspired by the success of reasoning models, we adopt a "Supervised Fine-Tuning (SFT) + Reinforcement Learning (RL)" pipeline to achieve AFRL. However, directly applying existing RL training often leads to \textbf{mode collapse} in the search relevance task, where the model forgets complex long-tail rules in pursuit of high rewards. From an information theory perspective: RL inherently minimizes the \textbf{Reverse KL divergence}, which tends to seek probability peaks (mode-seeking) and is prone to "reward hacking." On the other hand, SFT minimizes the \textbf{Forward KL divergence}, forcing the model to cover the data distribution (mode-covering) and effectively anchoring expert rules. Based on this insight, we propose a \textbf{Mode-Balanced Optimization} strategy, incorporating an SFT auxiliary loss into Stepwise-GRPO training to balance these two properties. Furthermore, we construct an automated instruction evolution system and a multi-stage curriculum to ensure expert-level data quality. Extensive experiments demonstrate that our 32B teacher model achieves state-of-the-art performance. Moreover, the AFRL architecture enables efficient knowledge distillation, successfully transferring expert-level logic to a 0.6B model, thereby reconciling reasoning depth with deployment latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。