解决强化学习中的隐藏混杂偏差，提升策略鲁棒性与泛化能力
Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Critic
- 基于后门准则构建可学习的伪历史变量重构器，实现观测数据下的因果干预估计
- 在连续控制任务中，相比基线方法，策略性能提升且对混杂因素更鲁棒
- 无需真实混杂变量或因果标签，适合存在隐含干扰的实际应用场景
隐藏混杂因子同时影响状态和动作，会误导强化学习中的策略学习，导致次优或不可泛化的行为。大多数强化学习算法忽略此问题，仅依赖统计关联学习策略。我们提出 DoSAC（基于后门调整的软演员-评论家），一种对 SAC 算法的严谨扩展，通过因果干预估计纠正隐藏混杂。DoSAC 利用后门准则估计干预策略 $π(a | \mathrm{do}(s))$，无需真实混杂变量或因果标签。为此，我们引入可学习的后门重构器，从当前状态推断伪过去变量（前一状态与动作），从而在观测数据中实现后门调整。该模块嵌入软演员-评论家框架，用于计算干预策略及其熵。在连续控制基准测试中，实验表明，当存在混杂时，DoSAC 显著优于基线方法，表现出更强的鲁棒性、泛化性和策略可靠性。
原文摘要 · Abstract (English)
Hidden confounders that influence both states and actions can bias policy learning in reinforcement learning (RL), leading to suboptimal or non-generalizable behavior. Most RL algorithms ignore this issue, learning policies from observational trajectories based solely on statistical associations rather than causal effects. We propose DoSAC (Do-Calculus Soft Actor-Critic with Backdoor Adjustment), a principled extension of the SAC algorithm that corrects for hidden confounding via causal intervention estimation. DoSAC estimates the interventional policy $π(a | \mathrm{do}(s))$ using the backdoor criterion, without requiring access to true confounders or causal labels. To achieve this, we introduce a learnable Backdoor Reconstructor that infers pseudo-past variables (previous state and action) from the current state to enable backdoor adjustment from observational data. This module is integrated into a soft actor-critic framework to compute both the interventional policy and its entropy. Empirical results on continuous control benchmarks show that DoSAC outperforms baselines under confounded settings, with improved robustness, generalization, and policy reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。