arXiv:2606.09165cs.AI2026-06中稿 · ICML

让安全评估模型更稳更准,通过渐进式训练适应不同评分标准。

Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges

论文配图:Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges
图 1 · 摘自论文原文
  • 用动态评分标准训练模型,模拟真实评分变化场景。
  • 跨评分标准准确率波动仅0.76,优于主流模型30%以上。
  • 适合需要稳定安全评估的AI系统部署与评测人员。

安全评估模型被广泛用于判断模型输出是否符合安全规范,但现有研究发现其在提示词和评分标准变化下仍显脆弱,仅风格扰动就导致假阴性率波动达0.24。我们认为安全判断本质是遵循评分标准的任务:鲁棒的评估器应在不同评分表述下保持一致。为此提出一种从可靠到灵活的渐进式训练策略:首先使用固定评分标准进行监督训练,再逐步引入由提示-响应-标签三元组生成的动态评分标准数据。在单一人工标注数据集上,针对三种不同评分标准(HarmBench式、ShieldGemma式及领域特定)进行评估,120亿参数的课程训练判别器在三种标准下均达到94.12%-94.88%准确率,跨标准波动仅为0.76,显著优于通用大模型、专用安全分类器和推理导向判别器,最高提升达30%。消融实验表明,直接混合动态评分数据会加剧波动(方差从1.44升至3.60),唯有采用渐进式课程才能恢复并超越固定评分基线(方差0.76)。

原文摘要 · Abstract (English)

Safety judges are increasingly deployed to evaluate model outputs against evolving criteria, yet recent meta-evaluation work shows they remain brittle under prompt and rubric variation, with false negative-rate swings of up to 0.24 reported for stylistic perturbations alone. We argue that safety judgment is fundamentally a rubric-following problem: a robust judge must apply the given evaluation criteria consistently across rubric formulations rather than memorize one specific template. We propose a training strategy that combines (i) instance-conditioned dynamic rubrics generated from prompt-response-label triples to expose the judge to the variability of evaluation criteria, and (ii) a reliable-to-expressive curriculum that begins with clean fixed-rubric supervision and progressively introduces noisier dynamic-rubric data. We evaluate on a single human-labeled set under three contrasting rubric prompts (HarmBench-style, ShieldGemma-style, and a domain-specific rubric). Our 12B curriculum judge achieves 94.12-94.88% accuracy across the three rubrics with a cross-rubric range of only 0.76, outperforming general-purpose LLMs, dedicated safety classifiers, and reasoning-oriented judges up to 30B in both peak accuracy and stability. An ablation shows that naively mixing dynamic rubrics into SFT increases cross rubric variance (1.44 -> 3.60); only the curriculum schedule recovers and improves on the fixed rubric baseline (variance 0.76).

安全评估评分标准模型训练鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。