自动融合人类设计的多个奖励函数，提升强化学习多目标任务表现。
Automatic Reward Shaping from Multi-Objective Human Heuristics
- 通过双层优化自动整合多种人工奖励，生成统一奖励函数。
- 在MuJoCo和Isaac Sim中实现与手动调优相当的任务性能。
- 引入噪声机制促进探索，避免陷入次优解。
在多目标强化学习环境中，设计有效的奖励函数仍是核心挑战。本文提出多目标奖励塑形与探索框架MORSE，可自动将多个由人类设计的启发式奖励整合为统一奖励函数。MORSE将塑形过程建模为双层优化问题：内层循环训练策略以最大化当前塑形奖励，外层循环更新奖励函数以优化任务表现。为促进奖励空间中的探索并避免次优局部极小值，MORSE引入随机性，通过固定且随机初始化的神经网络预测误差与任务表现引导噪声注入。在MuJoCo和Isaac Sim环境中的实验表明，MORSE能有效平衡多种目标，在多种机器人任务中达到与手动调优奖励函数相当的任务性能。
原文摘要 · Abstract (English)
Designing effective reward functions remains a central challenge in reinforcement learning, especially in multi-objective environments. In this work, we propose Multi-Objective Reward Shaping with Exploration (MORSE), a general framework that automatically combines multiple human-designed heuristic rewards into a unified reward function. MORSE formulates the shaping process as a bi-level optimization problem: the inner loop trains a policy to maximize the current shaped reward, while the outer loop updates the reward function to optimize task performance. To encourage exploration in the reward space and avoid suboptimal local minima, MORSE introduces stochasticity into the shaping process, injecting noise guided by task performance and the prediction error of a fixed, randomly initialized neural network. Experimental results in MuJoCo and Isaac Sim environments show that MORSE effectively balances multiple objectives across various robotic tasks, achieving task performance comparable to those obtained with manually tuned reward functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。