提出新方法识别强化学习中可控制的状态维度。
Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learning
- 用动作随机化作为干预手段，通过统计检验判断各观测维度是否可控。
- 在12个场景中，11个达到最优性能，优于传统观察方法。
- 适合状态含干扰项的复杂强化学习任务研究者使用。
当强化学习代理的观测包含与真实状态共享混杂因素的干扰项时，仅靠观测数据无法确定哪些维度受代理控制。在我们的基准测试中，即使采用状态条件的观测选择器，当干扰项模仿可控状态变量时仍会失效。我们提出干预边界发现（IBD），将代理自身的动作通道作为随机干预源：随机化动作实现干预对比，对每个维度进行两样本检验并校正错误发现率，生成观测维度的二值掩码。在包含最多100个干扰项的12个连续控制设置中，IBD在11个场景中达到与已知最优（oracle）回报相当的表现，而包括互信息、状态条件前向模型和基于梯度敏感性的观测基线方法，常因直接传递完整观测给SAC而表现更差。
原文摘要 · Abstract (English)
When an RL agent's observations contain distractors driven by the same confounders as its true state, observational data alone cannot identify which dimensions the agent controls. In our benchmarks, even state-conditioned observational selectors can collapse when distractors mimic controllable state variables. We propose Interventional Boundary Discovery (IBD), which treats the agent's own action channel as a source of randomized interventions: randomizing actions implements an interventional contrast, and per-dimension two-sample tests with FDR correction produce a binary mask over observation dimensions. Across 12 continuous-control settings with up to 100 distractors, IBD matches oracle return in 11 of 12 settings, while observational baselines including mutual information, state-conditioned forward models, and gradient-based sensitivity often underperform simply passing the full observation to SAC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。