解决离线强化学习中动作不可见的问题，让模型从噪声观测中准确评估策略效果。
Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
- 用下一状态作为隐藏动作的代理变量，实现策略价值的可识别性
- 提出多鲁棒估计器LURE，即使部分模型假设错误仍保持一致估计
- 首次解决带隐藏动作的离线强化学习问题，适合医疗等高噪声场景
标准离线强化学习算法通常假设数据集中的动作是无误差观测到的。然而在许多真实场景中，真实动作不可观测，仅有噪声代理可用，导致现有方法产生偏差和误导性结论。本文研究无限时域折扣马尔可夫决策过程下的离策略评估问题，当动作隐藏时，利用下一状态作为未观测动作的自然代理，建立了策略价值的可识别性，并提出一种基于影响函数的估计器LURE（Learning from the Unseen: Robust Estimator）。LURE具有多重鲁棒性，在多个扰动组件正确设定的情况下仍保持一致性，且渐近正态，支持有效的统计推断。据我们所知，这是首个解决隐藏动作下离线强化学习的工作。通过模拟实验和基于MIMIC-III数据库的脓毒症管理应用，验证了LURE的有效性。
原文摘要 · Abstract (English)
Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。