用自然语言规则替代神经网络参数，实现可解释的奖励建模
Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling
- 将奖励模型从权重优化转为自然语言规则搜索，提升可解释性
- 仅用70组偏好数据，效果超过全训练模型，在RewardBench2上达80.91%
- 适合需要透明、轻量级对齐机制的研究与应用
传统奖励建模依赖神经网络权重的梯度下降，形成数据密集、难以理解的“黑箱”。本文提出从隐式参数化转向显式参数化的新范式，将优化空间从连续权重转移到自然语言规则的离散空间。我们设计了一种无需训练的迭代规则学习框架：通过验证驱动的局部精炼生成判别性标准，并基于信息论编码率目标全局压缩候选标准池，形成紧凑的核心规则集。该核心集被组织为分层结构，高层评估维度由具体验证检查支撑，构成可解释、可迁移的奖励函数。实验表明，该方法挑战了数据规模依赖的假设：仅使用70组偏好对，所学规则指导的评判器在多个基准上超越全训练模型。例如，Qwen3-8B结合我们的规则在RewardBench2上达到80.91%，优于专用的Skywork-Reward-V2-Qwen3-8B（78.20%）。结果证明，对齐信号高度可压缩，可通过显式符号搜索有效捕捉。
原文摘要 · Abstract (English)
Conventional reward modeling relies on gradient descent over neural weights, creating opaque, data-hungry "black boxes." We propose a paradigm shift from implicit to explicit reward parameterization, recasting optimization from continuous weight spaces to the discrete space of natural language rubrics. We introduce a training-free framework based on iterative rubric learning: it locally induces discriminative criteria via verification-driven refinement, and globally compresses the candidate criteria pool into a compact core set by maximizing an information-theoretic coding rate objective. We organize the compressed core set into a hierarchical rubric structure -- high-level evaluation dimensions supported by concrete verification checks -- serving as an interpretable, portable reward function. Empirically, our approach challenges prevailing data scaling assumptions: using only 70 preference pairs, our rubric-guided judges outperform fully trained reward models on diverse benchmarks. For instance, Qwen3-8B equipped with our learned rubrics achieves 80.91% on RewardBench2, surpassing the specialized Skywork-Reward-V2-Qwen3-8B (78.20%). These results demonstrate that alignment signals are highly compressible and can be effectively captured through explicit symbolic search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。