通过惩罚奖励方差，让模型不再忽视困难约束，提升多目标对齐可靠性。
RVPO: Risk-Sensitive Alignment via Variance Regularization
- 用方差惩罚替代平均聚合，让模型关注所有目标的一致性表现。
- 在17个奖励信号下，健康推理任务得分提升至0.261（对比基线0.215，显著提升）。
- 适合需要严格遵守安全或格式等硬约束的场景，如医疗与科学推理。
当前无评判器的强化学习人类反馈方法通过算术平均聚合多目标奖励，易导致约束忽视：单一目标的高分可掩盖其他关键目标（如安全或格式）的严重失败，从而隐藏影响可靠性的‘瓶颈’奖励。我们提出风险敏感的奖励-方差策略优化（RVPO），在优势值聚合阶段惩罚不同奖励间的方差，将目标从‘最大化总和’转变为‘最大化一致性’。通过泰勒展开证明，LogSumExp（SoftMin）操作符能有效实现平滑的方差惩罚。我们在基于评分标准的医疗与科学推理任务中评估了RVPO，使用最多17个并发的LLM评分奖励信号（Qwen2.5-3B/7B/14B），并在基于规则的工具调用任务中测试（Qwen2.5-1.5B/3B）。RVPO防止模型忽略困难约束而只追求简单目标，使HealthBench得分达到0.261（14B模型，相比GDPO的0.215，p < 0.001），同时在GPQA-Diamond上保持竞争力准确率，且未出现其他多奖励方法常见的后期性能下降，表明方差正则化在不同模型规模下均能缓解约束忽视，且不损害通用能力。
原文摘要 · Abstract (English)
Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing "bottleneck" rewards vital for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation, shifting the objective from "maximize sum" to "maximize consistency." We show via Taylor expansion that a LogSumExp (SoftMin) operator effectively acts as a smooth variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 concurrent LLM-judged reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). By preventing the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench (0.261 vs. 0.215 for GDPO at 14B, $p < 0.001$) and maintains competitive accuracy on GPQA-Diamond without the late-stage degradation observed in other multi-reward methods, demonstrating that variance regularization mitigates constraint neglect across model scales without sacrificing general capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。