让AI自动生成评分标准,边写边优化,提升开放生成质量。
EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

- 用同一模型交替生成内容和评分标准,实现动态协同优化。
- 在医疗、写作、科学领域均超越静态与外部模型驱动的方法。
- 适合需要持续改进生成质量的开放任务,如创意写作或专业问答。
强化学习(RL)在可验证领域显著提升了大语言模型(LLM)性能,但开放生成任务因缺乏明确奖励信号而难以对齐。现有基于评分标准的RL方法依赖人工静态标注的标准,导致策略滞后;或使用昂贵的外部专用模型进行动态更新。本文提出EvoRubric,一种单策略协同进化强化学习框架,无需静态标准或外部评分生成器。通过将响应生成与评分标准生成统一于单一参数化策略中,EvoRubric动态交替执行推理者(Reasoner)与评分生成器(Rubric Generator)角色。为防止奖励滥用并确保信号可靠性,引入多层级验证机制:元验证器、零方差剪枝及留一法同伴共识。经验证的标准动态存入记忆池,形成密集、多目标奖励信号,持续协同优化双角色。跨医学、写作、科学领域的实验表明,EvoRubric始终优于传统静态与外部大模型驱动的对齐方法。值得注意的是,该框架兼容人类专家先验知识;当以专家标注标准初始化时,EvoRubric能进一步发现新颖判别维度,性能优于仅依赖静态专家标注的情况。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has significantly advanced Large Language Models (LLMs) in verifiable domains, but aligning models for open-ended generation remains profoundly challenging due to the lack of definitive rewards. Current rubric-based RL methods mitigate this by employing explicit criteria; however, they rely heavily on static, human-annotated rubrics that inevitably cause policy lag, or expensive external proprietary models for dynamic updates. In this paper, we propose EvoRubric, a novel single-policy co-evolutionary RL framework that eliminates the reliance on static criteria and on external rubric generators. By unifying response generation and rubric generation under a single parameterized policy, EvoRubric dynamically alternates between a Reasoner and a Rubric Generator. To prevent reward hacking and ensure the reliability of generated signals, we introduce a multi-level verification pipeline featuring a meta-verifier, zero-variance pruning, and a Leave-One-Out peer consensus mechanism. Validated criteria are dynamically archived into a memory pool, yielding dense, multi-objective rewards to continuously co-optimize both roles. Extensive experiments across Medical, Writing, and Science domains demonstrate that EvoRubric consistently outperforms traditional static and external-LLM-driven alignment methods. Notably, our framework is compatible with human-expert priors. When initialized with expert-annotated rubrics, EvoRubric can further uncover novel, discriminative dimensions, achieving better performance than relying solely on static expert annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。