arXiv:2608.24231cs.CL2026-08

让大模型自己评自己,实现低成本持续优化评分能力。

RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges

论文配图:RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges
图 1 · 摘自论文原文
  • 构建双阶段自评估框架,模型先打分再自我审核,避免评分作弊。
  • 在多个基准测试中显著提升评分准确性,效果优于外部监督或教师模型。
  • 适合需要低成本高质量评分的AI对齐与评测场景。

大模型作为评分员在开放文本评估和后训练引导中至关重要,但通常依赖昂贵的人工标注、奖励模型或强教师模型进行优化。本文提出闭环式有界递归自改进(RecurSE):模型自身生成评价信号用于训练,无需外部黄金标注。核心设计包含两个阶段:第一阶段,可训练评分器依据规则评估回复;第二阶段,同步的策略复制检查器通过元规则审计评分器推理过程,生成标量过程奖励。通过结构解耦,隔离检查器得分与评分器输出,消除因令牌复制导致的虚假奖励膨胀。此外,成对优势有效性(PAV)作为无偏验证监控器,联合追踪评分准确率与检查器忠实度,可靠识别最优早停窗口。在Qwen3.5-9B、Gemma-4-E4B-it和Qwen3.6-27B上,RecurSE在医疗、成对比较、摘要与专业任务等保留测试集上均实现一致泛化提升。消融实验表明,评分器与检查器协同进化优于固定检查器、外部元评分器、自一致性及教师蒸馏。此外,由该评分器筛选的偏好对有效增强下游策略对齐。因此,当自产奖励有效性被显式解耦并监控时,大模型自评的有界递归改进是可行的。

原文摘要 · Abstract (English)

LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from stronger teachers. In this work, we eliminate external gold supervision from the RL training reward: the model's own evaluative capability generates learning signals for its optimization -- a closed-loop setting of bounded recursive self-improvement (RSI) termed Recursive Self-Evaluation (RecurSE). We study two central questions: when can self-improvement occur, and when must it stop? First, RecurSE pairs a trainable judge evaluating candidate responses under per-rule rubrics (Pass 1) with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). To enable learning, interface decoupling structurally isolates the checker's scalar score from the judge's verdict tokens, eliminating a degenerative token-copying shortcut that inflates self-assigned rewards. Second, because unanchored recursive learning is inherently bounded, Pairwise Advantage Validity (PAV) serves as an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Ablations demonstrate that synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Furthermore, preference pairs curated by our judge effectively enhance downstream policy alignment. Bounded RSI for LLM-as-judge is thus viable when self-produced reward validity is explicitly decoupled and monitored.

大模型评分自评估强化学习对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。