通过因果效应估计减少冗余动作，提升强化学习探索效率。
Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation
- 用逆动力学模型预训练，量化每步动作的因果影响。
- 在仿真环境中验证，显著减少重复无效尝试。
- 适合需要高效探索的复杂决策任务研究者。
深度强化学习在庞大且冗余的动作空间中进行智能决策仍具挑战性。若每一步存在相似但无效的动作，会导致重复且低效的试错。现有方法尝试通过减少或惩罚冗余动作来改进探索，但缺乏量化且可靠的冗余判断依据。本文提出一种基于动作因果效应估计的方法，以提升探索效率。不同于以往方法，本方法能为单步转移提供动作因果关系的定量分析。首先预训练逆动力学模型作为环境先验知识；随后在每一步对全动作空间进行分类，并估计各动作的因果效应，从而在探索中抑制冗余动作。我们提供了理论分析证明该方法的有效性，并在含冗余动作的仿真环境中进行实证评估。代码已开源：https://github.com/agi-brain/cee.git。
原文摘要 · Abstract (English)
Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method and present empirical results from simulations in environments with redundant actions to evaluate its performance. Our implementation is available at https://github.com/agi-brain/cee.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。