考虑感知成本的决策问题，智能体需权衡状态感知代价与行动收益。
MDPs with a State Sensing Cost
- 将感知成本纳入马尔可夫决策过程，构建扩展状态空间模型
- 推导最优值函数下界，可量化任意策略的性能差距
- 提出高效算法SPI，实测接近最优，适合资源受限场景
在许多实际序列决策问题中，追踪环境状态会产生感知、通信或计算成本。在此类场景中，智能体与环境的交互需额外决定何时感知状态，以平衡状态特定最优动作的价值与感知成本。本文将其建模为带有期望折扣成本的马尔可夫决策过程（MDP），智能体在感知下一状态时需支付额外成本，但也可选择在‘盲态’下执行动作。我们将该问题形式化为经典折扣成本MDP，其状态空间被扩展为可数无穷大。尽管一般情况下最优策略难以计算，我们推导了最优值函数的下界，从而能够界定任意策略的次优性差距。此外，我们提出一种基于策略改进的计算高效算法SPI，实践中表现接近最优策略。最后通过数值案例对比了当前最先进方法。
原文摘要 · Abstract (English)
In many practical sequential decision-making problems, tracking the state of the environment incurs a sensing/communication/computation cost. In these settings, the agent's interaction with its environment includes the additional component of deciding when to sense the state, in a manner that balances the value associated with optimal (state-specific) actions and the cost of sensing. We formulate this as an expected discounted cost Markov Decision Process (MDP), wherein the agent incurs an additional cost for sensing its next state, but has the option to take actions while remaining `blind' to the system state. We pose this problem as a classical discounted cost MDP with an expanded (countably infinite) state space. While computing the optimal policy for this MDP is intractable in general, we derive lower bounds on the optimal value function, which allow us to bound the suboptimality gap of any policy. We also propose a computationally efficient algorithm SPI, based on policy improvement, which in practice performs close to the optimal policy. Finally, we benchmark against the state-of-the-art via a numerical case study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。