arXiv:2606.07705cs.LGcs.AI2026-06被引 1

动态调整奖励权重,让大模型对不同目标的学习更均衡高效。

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

论文配图:SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models
图 1 · 摘自论文原文
  • 用变异系数衡量各目标信息量,实时动态分配奖励权重。
  • 在工具调用和摘要任务上,训练效率与最终性能均显著提升。
  • 无需额外计算开销,可直接插入现有强化学习框架使用。

尽管多目标强化学习(MORL)是使大语言模型符合复杂人类偏好的核心,但当前普遍采用的静态加权求和忽略了奖励学习在各目标间存在显著异步性。已充分学习的目标产生同质化、低方差信号,其残余噪声会污染聚合奖励(在GRPO中)或占用固定优势预算份额(在GDPO中),干扰尚未充分学习目标所携带的稀有高价值信号。为此,我们提出阶段感知动态加权(SAW),一种轻量级、算法无关的动态加权机制。SAW以变异系数(CV)作为无量纲的信息量代理,根据批次内各维度相对信息量重新加权其奖励或优势贡献。与依赖多次前向与反向传播的梯度方法不同,SAW仅基于批次统计,计算开销几乎可忽略。在工具调用和文本摘要任务上的实验表明,SAW在GRPO与GDPO框架下均能持续提升训练效率与最终性能,证实其为通用型多奖励大模型对齐插件。代码已开源于 https://github.com/Zhaolutuan/SAW。

原文摘要 · Abstract (English)

Although multi-objective reinforcement learning (MORL) is central to aligning large language models with complex human preferences, the prevailing practice of static weighted summation overlooks a more fundamental phenomenon: reward learning is markedly asynchronous across objectives. Well-learned dimensions quickly produce homogeneous, low-variance signals whose residual noise contaminates the aggregated reward (in GRPO) or occupies a fixed share of the advantage budget (in GDPO), interfering with the scarce yet high-value signals carried by under-learned dimensions. To address this asynchrony, we propose Stage-Aware Dynamic Weighting (SAW), a lightweight, algorithm-agnostic dynamic weighting mechanism. SAW utilizes the coefficient of variation (CV) as a scale-invariant proxy for real-time informativeness, reweighting each dimension's reward or advantage contribution by its relative informativeness within the batch. Unlike gradient-based methods that require multiple forward and backward passes, SAW relies solely on batch-level statistics, introducing nearly negligible computational overhead. Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance under both GRPO and GDPO frameworks, confirming it as a general-purpose plug-in for multi-reward LLM alignment. Our code is available at https://github.com/Zhaolutuan/SAW

强化学习多目标优化大模型对齐动态加权

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。