让大模型自己生成评价标准,实现无监督自我进化。
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

- 用模型自身输出构建可区分的评价标准,替代人工标注。
- 在RewardBench-2上超越GPT-4.1 25.7%,在OLMo3-Adapt上达69.3%。
- 无需外部数据或人工干预,适合追求自驱动优化的研究者。
语言模型在预训练中蕴含大量评估知识,但现有后训练方法依赖外部监督(如人工标注、专有模型或标量奖励模型)生成奖励信号,各自存在局限:人类判断无法覆盖自身能力之外的领域,专有API带来依赖性,可验证奖励仅适用于有真值答案的领域。利用模型自身评估能力作为奖励来源,可随模型规模扩展,但尚未被充分利用。本文提出EVOLM,将模型的评估能力结构化为显式的判别性评价标准,并以此作为训练信号。EVOLM在单个语言模型中交替训练两种能力:(1) 生成器负责产出针对具体实例的评价标准,以最大化一个固定裁判模型区分优选与劣质回答的能力;(2) 策略模型则使用这些标准条件下的得分作为奖励进行训练。所有偏好信号均通过策略模型自身输出与早期检查点的时间对比生成,无需人工标注或外部监督。在Qwen3-8B上,EVOLM生成的评价标准在RewardBench-2上优于GPT-4.1 25.7%;联合训练的策略在OLMo3-Adapt套件上平均得分69.3%,优于使用GPT-4.1提示标准训练的策略3.9%,也优于当前最先进的8B奖励模型SkyWork-RM 16%。结果表明,将模型评估能力结构化为协同进化的判别性标准,可实现无需外部监督的自我提升。
原文摘要 · Abstract (English)
Language models encode substantial evaluative knowledge from pretraining, yet current post-training methods rely on external supervision (human annotations, proprietary models, or scalar reward models) to produce reward signals. Each imposes a ceiling. Human judgment cannot supervise capabilities beyond its own, proprietary APIs create dependencies, and verifiable rewards cover only domains with ground-truth answers. Self-improvement from a model's own evaluative capacity is a reward source that scales with the model itself, yet remains largely untapped by current methods. We introduce EVOLM, a post-training method that structures this capacity into explicit discriminative rubrics and uses them as training signal. EVOLM trains two capabilities within a single language model in alternation: (1) a rubric generator producing instance-specific evaluation criteria optimized for discriminative utility, which maximizes a small frozen judge's ability to distinguish preferred from dispreferred responses; and (2) a policy trained using those rubric-conditioned scores as reward. All preference signals are constructed from the policy's own outputs via temporal contrast with earlier checkpoints, requiring no human annotation or external supervision. EVOLM trains a Qwen3-8B model to generate rubrics that outperform GPT-4.1 on RewardBench-2 by 25.7%. The co-trained policy achieves 69.3% average on the OLMo3-Adapt suite, outperforming policies trained with GPT-4.1 prompted rubrics by 3.9% and with the state-of-the-art 8B reward model SkyWork-RM by 16%. Overall, EVOLM demonstrates that structuring a model's evaluative capacity into co-evolving discriminative rubrics enables self-improvement without external supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。