让智能体学会在看与不动之间权衡,省钱又高效。
OCMDP: Observation-Constrained Markov Decision Process
- 将观察和控制分开学,动态决定何时该看、看什么。
- 在模拟诊断和真实医疗场景中,观察成本平均降低显著。
- 适合资源受限的实时决策系统,如医疗、工业监控。
在许多实际应用中,决策需权衡信息获取成本与收益。传统控制系统假设状态完全可观,但当观测代价高昂时这不现实。本文提出观察约束马尔可夫决策过程(OCMDP),让策略同时影响观测行为与控制动作。为应对观测与控制联合动作带来的复杂性,设计了一种迭代的无模型深度强化学习算法,将策略分解为感知与控制两部分。该分解使算法能在无需环境动力学知识的情况下,高效学习何时以及如何观测,并制定最优控制动作。在模拟诊断任务和基于HeartPole的真实医疗环境中验证表明,本方法平均显著降低了观测成本,效率明显优于基线方法。
原文摘要 · Abstract (English)
In many practical applications, decision-making processes must balance the costs of acquiring information with the benefits it provides. Traditional control systems often assume full observability, an unrealistic assumption when observations are expensive. We tackle the challenge of simultaneously learning observation and control strategies in such cost-sensitive environments by introducing the Observation-Constrained Markov Decision Process (OCMDP), where the policy influences the observability of the true state. To manage the complexity arising from the combined observation and control actions, we develop an iterative, model-free deep reinforcement learning algorithm that separates the sensing and control components of the policy. This decomposition enables efficient learning in the expanded action space by focusing on when and what to observe, as well as determining optimal control actions, without requiring knowledge of the environment's dynamics. We validate our approach on a simulated diagnostic task and a realistic healthcare environment using HeartPole. Given both scenarios, the experimental results demonstrate that our model achieves a substantial reduction in observation costs on average, significantly outperforming baseline methods by a notable margin in efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。