arXiv:2606.03361cs.LG2026-06

解决评分奖励错误传播问题,让评分更符合逻辑依赖关系。

Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning

论文配图:Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning
图 1 · 摘自论文原文
  • 构建带依赖关系的评分图模型,用概率传播抑制不合法奖励
  • 在三个评测集上提升效果,最高比传统方法高15.5%
  • 适合需要严谨评分机制的开放文本生成任务

基于评分标准的奖励在开放文本语言模型后训练中越来越常用,但当前常将各评分维度独立处理,忽略了评分项之间的前提与触发关系,导致条件不满足时仍可能获得奖励或惩罚。这种结构化聚合失效现象称为‘错误信用传播’(FCP)。为此,我们提出GEAR(图形事件聚合用于评分奖励),一种依赖感知的评分聚合概率图框架。GEAR将每个评分项视为类型化评分图中的潜在伯努利事件,从无支持的父事件向子事件传播软抑制,并将结果事件概率归一化为期望带符号效用。该方法实现线性时间计算,可直接嵌入标准评分强化学习流程而无需改变优化算法。在HealthBench、WritingBench和PLawBench三个数据集上,使用两种策略模型验证表明,GEAR持续优于扁平聚合和确定性门控,相对提升最高达15.5%。FCP诊断显示,相比扁平聚合,其泄漏降低96.5%,同时保留更多合法下游效用,优于确定性门控。代码已公开于https://github.com/LvCan926/GEAR。

原文摘要 · Abstract (English)

Rubric-based rewards are increasingly used for open-ended language model post-training, but criterion-level scores are often aggregated as independent utilities. This flat scalarization ignores rubric-specified prerequisite and activation relations among criteria, allowing reward or penalty to be counted even when the condition that licenses it is absent. We call this structural reward-aggregation failure \textbf{False Credit Propagation} (FCP). To address this limitation, we propose \ourname (\textbf{G}raphical \textbf{E}vent \textbf{A}ggregation for \textbf{R}ubric rewards), a probabilistic graphical framework for dependency-aware rubric aggregation. \ourname models each criterion outcome as a latent Bernoulli event in a typed rubric graph, propagates soft suppression from unsupported parent events to their children, and aggregates the resulting event probabilities into a normalized expected signed utility. This yields a linear-time reward computation that can be plugged into standard rubric-based RL pipelines without changing the outer optimization algorithm. Experiments on HealthBench, WritingBench, and PLawBench with two policy backbones show that \ourname consistently improves over flat aggregation and deterministic gating, achieving relative gains of up to 15.5\% over flat aggregation. FCP diagnostics further show that \ourname reduces leakage by 96.5\% relative to flat aggregation while preserving more licensed downstream utility than deterministic gating. Our code is publicly available at https://github.com/LvCan926/GEAR.

强化学习评分系统概率图语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。