提出图模型方法，解决离线强化学习中状态变量选择的因果识别问题。
A Graphical Approach to State Variable Selection in Off-policy Learning
- 基于无环有向混合图（ADMG）构建状态变量选择的图形化识别准则。
- 在集装箱物流动态定价模拟中验证：违反准则会导致次优策略。
- 澄清了医疗与强化学习领域对因果假设的常见误解，适用于复杂决策系统。
序列决策问题在科学多个领域广泛研究。从历史数据中学习策略——即离线学习——面临的核心挑战是：当观测数据非随机时，如何准确识别目标策略的影响。离线学习主要研究两个场景：动态治疗方案（DTRs），关注医疗短周期决策中的混杂控制；以及离线强化学习（RL），关注封闭系统如游戏中的维度压缩。这两个成熟领域的差距限制了离线学习在现实问题中的广泛应用。本文基于无环有向混合图（ADMG）的因果推断理论，为一般决策过程提供了一套图形化识别准则，涵盖DTRs和马尔可夫决策过程（MDPs）。我们讨论了这些结果与现有DTR和RL文献中常隐含的因果假设的关系，并澄清了几种常见误解。最后，通过一个真实的集装箱物流动态定价问题的模拟研究，展示了违背我们的图形准则可能导致次优策略。
原文摘要 · Abstract (English)
Sequential decision problems are widely studied across many areas of science. A key challenge when learning policies from historical data - a practice commonly referred to as off-policy learning - is how to ``identify'' the impact of a policy of interest when the observed data are not randomized. Off-policy learning has mainly been studied in two settings: dynamic treatment regimes (DTRs), where the focus is on controlling confounding in medical problems with short decision horizons, and offline reinforcement learning (RL), where the focus is on dimension reduction in closed systems such as games. The gap between these two well studied settings has limited the wider application of off-policy learning to many real-world problems. Using the theory for causal inference based on acyclic directed mixed graph (ADMGs), we provide a set of graphical identification criteria in general decision processes that encompass both DTRs and MDPs. We discuss how our results relate to the often implicit causal assumptions made in the DTR and RL literatures and further clarify several common misconceptions. Finally, we present a realistic simulation study for the dynamic pricing problem encountered in container logistics, and demonstrate how violations of our graphical criteria can lead to suboptimal policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。