从离线数据自动学习鲁棒奖励函数，提升连续控制的因果稳定性
Confounding Robust Continuous Control via Automatic Reward Shaping
- 基于因果贝尔曼方程，从含混淆变量的离线数据中学习奖励塑形函数
- 在多个基准上使用SAC算法验证，对未观测混淆变量具有强鲁棒性
- 适合关注因果强化学习与真实场景鲁棒性的研究者参考
奖励塑形被广泛用于加速强化学习（RL）代理的训练。然而，针对复杂连续控制问题，设计有效的奖励塑形函数仍缺乏系统方法。本文提出一种从含未观测混淆变量的离线数据中自动学习奖励塑形函数的方法。具体地，该方法基于最近提出的因果贝尔曼方程，学习最优状态值的紧上界，并将其作为潜在函数用于基于势能的奖励塑形（PBRS）框架。所提算法在多个常用连续控制基准上与Soft-Actor-Critic（SAC）结合测试，展现出在未观测混淆变量下的强性能保障。本工作标志着从因果视角实现混淆鲁棒连续控制的重要第一步。训练代码见：https://github.com/mateojuliani/confounding_robust_cont_control。
原文摘要 · Abstract (English)
Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。