arXiv:2410.03892cs.LGcs.AI2024-10被引 2

让智能体主动获取信息做决策,兼顾成本与收益。

Towards Cost Sensitive Decision Making

  • 设计可主动采集特征的强化学习框架,动态权衡信息成本与决策收益。
  • 在多个任务上表现优于传统部分可观测强化学习方法,提升决策质量。
  • 适合需实时决策且信息获取有代价的应用场景,如医疗诊断、自动驾驶。

许多现实场景中,决策时面对有限或不确定的数据,可主动获取额外信息。然而,传统强化学习方法要么要求所有特征预先获取(如MDP),要么将部分特征视为不可获取的缺失数据(如POMDP)。本文提出主动采集部分可观测马尔可夫决策过程(AA-POMDP),针对不同应用场景识别两类采集机制。为辅助智能体在主动采集环境中做出判断并缓解探索-利用困境,我们采用基于模型的方法,使用深度生成模型捕捉特征间依赖关系,并对未观测特征进行补全,补全结果即代表智能体的信念。结合动态模型,我们设计了分层强化学习算法,解决两类AA-POMDP问题。实验表明,该方法显著优于现有POMDP强化学习方案。

原文摘要 · Abstract (English)

Many real-world situations allow for the acquisition of additional relevant information when making decisions with limited or uncertain data. However, traditional RL approaches either require all features to be acquired beforehand (e.g. in a MDP) or regard part of them as missing data that cannot be acquired (e.g. in a POMDP). In this work, we consider RL models that may actively acquire features from the environment to improve the decision quality and certainty, while automatically balancing the cost of feature acquisition process and the reward of task decision process. We propose the Active-Acquisition POMDP and identify two types of the acquisition process for different application domains. In order to assist the agent in the actively-acquired partially-observed environment and alleviate the exploration-exploitation dilemma, we develop a model-based approach, where a deep generative model is utilized to capture the dependencies of the features and impute the unobserved features. The imputations essentially represent the beliefs of the agent. Equipped with the dynamics model, we develop hierarchical RL algorithms to resolve both types of the AA-POMDPs. Empirical results demonstrate that our approach achieves considerably better performance than existing POMDP-RL solutions.

强化学习决策优化主动学习部分可观测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。