解决多目标强化学习中奖励频率差异导致的训练偏差问题。
PRISM: Parallel Reward Integration with Symmetry for MORL
- 通过反射对称性约束，统一不同频率的奖励通道。
- 在MuJoCo上实现超100%的超体积提升，优于稀疏奖励基线和全密集奖励模型。
- 适合研究多目标强化学习与奖励设计的学者使用。
本文研究异质性多目标强化学习（MORL），其中目标在时间频率上差异显著。这种异质性导致密集奖励主导学习过程，而稀疏的长周期奖励难以获得有效信用分配，从而降低样本效率。为此，提出并行奖励融合对称性算法（PRISM），引入基于理论的ReSymNet模型，利用残差结构学习缩放的机会值，加速探索同时保持最优策略。还设计了对称正则化器SymReg，强制代理镜像行为，将策略搜索限制在反射等变子空间内，理论上降低假设复杂度并提升泛化能力。在MuJoCo基准测试中，PRISM持续优于稀疏奖励基线和全密集奖励的基准模型，帕累托覆盖范围与分布平衡均有显著改善：相比基线超体积提升超过100%，相较基准模型最高达32%。代码已开源。
原文摘要 · Abstract (English)
This work studies heterogeneous Multi-Objective Reinforcement Learning (MORL), where objectives can differ sharply in temporal frequency. Such heterogeneity allows dense objectives to dominate learning, while sparse long-horizon rewards receive weak credit assignment, leading to poor sample efficiency. We propose a Parallel Reward Integration with Symmetry (PRISM) algorithm that enforces reflectional symmetry as an inductive bias in aligning reward channels. PRISM introduces ReSymNet, a theory-motivated model that reconciles temporal-frequency mismatches across objectives, using residual blocks to learn a scaled opportunity value that accelerates exploration while preserving the optimal policy. We also propose SymReg, a reflectional equivariance regulariser that enforces agent mirroring and constrains policy search to a reflection-equivariant subspace. This restriction provably reduces hypothesis complexity and improves generalisation. Across MuJoCo benchmarks, PRISM consistently outperforms both a sparse-reward baseline and an oracle trained with full dense rewards, improving Pareto coverage and distributional balance: it achieves hypervolume gains exceeding 100\% over the baseline and up to 32\% over the oracle. The code is at \href{https://github.com/EVIEHub/PRISM}{https://github.com/EVIEHub/PRISM}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。