用世界模型中的时间信息改进稀疏奖励下的目标导向强化学习信用分配。
Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
- 基于最优传输从占据度量模型中提取世界几何结构，生成新奖励函数。
- 在13个长序列任务上性能提升2.2倍，且不改变最优策略。
- 已在3个托卡马克控制任务中实现实体世界应用，适合复杂控制场景。
动作与长期后果之间的时间延迟使得从数据中学习目标导向行为时信用分配成为挑战。生成式世界模型捕捉了代理可能访问的未来状态分布，表明它们已蕴含时间信息。如何从中提取这些时间信息以实现信用分配？本文形式化地阐述了世界模型中存储的时间信息如何编码世界的潜在几何结构。利用最优传输，我们从学习到的占据度量模型中提取该几何结构，并构建一个捕捉目标达成信息的奖励函数。由此提出的方法——占据度量奖励塑造（Occupancy Reward Shaping, ORS），显著缓解了稀疏奖励设置下的信用分配问题。ORS严格保持最优策略不变，但在13个多样化的长时序运动与操作任务上实现了2.2倍的性能提升。此外，我们在真实世界中验证了其在3个托卡马克控制任务上的有效性。代码：https://github.com/aravindvenu7/occupancy_reward_shaping；网站：https://aravindvenu7.github.io/website/ors/
原文摘要 · Abstract (English)
The temporal lag between actions and their long-term consequences makes credit assignment a challenge when learning goal-directed behaviors from data. Generative world models capture the distribution of future states an agent may visit, indicating that they have captured temporal information. How can that temporal information be extracted to perform credit assignment? In this paper, we formalize how the temporal information stored in world models encodes the underlying geometry of the world. Leveraging optimal transport, we extract this geometry from a learned model of the occupancy measure into a reward function that captures goal-reaching information. Our resulting method, Occupancy Reward Shaping, largely mitigates the problem of credit assignment in sparse reward settings. ORS provably does not alter the optimal policy, yet empirically improves performance by 2.2x across 13 diverse long-horizon locomotion and manipulation tasks. Moreover, we demonstrate the effectiveness of ORS in the real world for controlling nuclear fusion on 3 Tokamak control tasks. Code: https://github.com/aravindvenu7/occupancy_reward_shaping; Website: https://aravindvenu7.github.io/website/ors/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。