arXiv:2607.20083cs.LGcs.AI2026-07被引 2

让评估器和模型一起进化,提升大模型生成质量。

Co-Evolving LLM Evaluators and Policies via DynamicRubric

论文配图:Co-Evolving LLM Evaluators and Policies via DynamicRubric
图 1 · 摘自论文原文
  • 基于响应集动态生成评分条目,实现评估与模型协同优化
  • 在8B模型上优于70B奖励模型和235B静态评分生成器
  • 已部署于微信搜索,日均处理千万级请求,显著提升线上指标

通过在策略生成的样本上进行评估反馈来改进大语言模型,已成为关键方法。随着策略优化,生成响应质量趋近,导致评估分数差距缩小,削弱了策略优化的监督信号。本文从概率分配视角理论分析发现,响应间分数差即为概率转移方向性的增益来源,揭示相对分数差是策略优化的核心信号。受此启发,提出DynamicRubric框架:根据候选响应集生成加权二元评分条目,并聚合为响应级得分。在8B骨干模型上的实验表明,DynamicRubric在评估性能和策略监督强度上均优于使用70B奖励模型或235B静态评分生成器的基线。经DynamicRubric优化的策略在可验证推理与编码任务中表现更优。其优化模型已在微信搜索的AI问答场景中全量上线,服务每日数千万请求,显著改善关键线上指标。结果表明,评估器应随其所监督的策略共同演化。

原文摘要 · Abstract (English)

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.

大模型优化评估器进化协同训练微信搜索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。