提出因果方法，让强化学习在数据有隐藏干扰时仍能稳定表现。
Confounding Robust Deep Reinforcement Learning: A Causal Approach
- 基于因果推断，寻找最坏情况下的安全策略
- 在12个受干扰的Atari游戏上均优于标准DQN
- 适合处理数据偏差严重的复杂决策场景
人工智能的关键任务是在未知环境中学习有效策略以优化性能。离策略学习方法（如Q-learning）允许智能体基于历史经验做出最优决策。本文研究在复杂高维环境中，存在无法事先排除的未观测混淆因素时，如何从有偏数据中进行离策略学习。基于广受认可的深度Q网络（DQN），我们提出一种对混淆偏差鲁棒的新算法。该算法旨在寻找与观测数据兼容的最坏情况环境下的安全策略。我们在12个受混淆的Atari游戏中应用该方法，结果表明，在行为策略与目标策略输入不匹配且存在未观测混淆因素的情况下，该方法在所有游戏中均持续优于标准DQN。
原文摘要 · Abstract (English)
A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where \emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。