解决多目标强化学习中的奖励冲突与震荡问题，提升训练稳定性和性能。
A Better Spur Should Start From Each Objective
- 从数据、梯度、约束三方面精细干预，避免传统线性标量化带来的优化冲突。
- 在真实电商数据集上显著提升训练稳定性，多个冲突指标同时优化效果更优。
- 适用于工具调用和代码生成等复杂任务，适合需要多目标对齐的实际场景。
现实世界中的多目标强化学习常面临奖励稀疏、奖励冲突及后期奖励拉扯问题，导致传统线性标量化方法出现严重指标振荡。为应对实际部署中多目标间的优化冲突，我们提出多边际偏好优化（MMPO），一种在数据、梯度和约束层面进行细粒度干预的框架，而非依赖粗粒度全局标量化。具体而言，MMPO通过暴露去偏缓解稀疏且有偏的奖励，采用优先级感知正交投影解耦冲突梯度，并引入自提示梯度约束防止主导目标压制弱目标。在真实电商数据集上的实验表明，MMPO提升了训练稳定性，并在多个冲突指标上持续取得更优表现。此外，其在ToolRL和代码生成等更广泛任务中也展现出良好泛化能力，证明其作为多目标对齐实用范式的有效性。
原文摘要 · Abstract (English)
Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。