提出DBMM模型,实现无需先验模型的高效部分可观测决策推断。
Deep Belief Markov Models for POMDP Inference
- 用变分推断构建深度信念马尔可夫模型,从观测数据中学习系统动态
- 在离散与连续变量的基准问题上实现模型无关的信念推断
- 适用于高维、非线性、无真实状态标签的复杂决策场景
本文提出一种新型深度学习架构——深度信念马尔可夫模型(DBMM),用于在部分可观测马尔可夫决策过程(POMDP)问题中实现高效、模型无关的推断。传统基于精确计算或采样算法的方法在高维、部分可观测环境中难以扩展,且常缺乏真实状态用于学习转移动态。DBMM将深度马尔可夫模型扩展至部分可观测决策框架,通过变分推断完全基于观测数据进行信念推断。利用神经网络的强大表达能力,DBMM可捕捉系统动态中的非线性关系,并自然适应高维及离散/连续变量问题。同时,神经网络参数可随数据动态高效更新。实验表明,该方法在包含离散与连续变量的基准任务中,具备良好的模型无关推断能力。
原文摘要 · Abstract (English)
This work introduces a novel deep learning-based architecture, termed the Deep Belief Markov Model (DBMM), which provides efficient, model-formulation agnostic inference in Partially Observable Markov Decision Process (POMDP) problems. The POMDP framework allows for modeling and solving sequential decision-making problems under observation uncertainty. In complex, high-dimensional, partially observable environments, existing methods for inference based on exact computations (e.g., via Bayes' theorem) or sampling algorithms do not scale well. Furthermore, ground truth states may not be available for learning the exact transition dynamics. DBMMs extend deep Markov models into the partially observable decision-making framework and allow efficient belief inference entirely based on available observation data via variational inference methods. By leveraging the potency of neural networks, DBMMs can infer and simulate non-linear relationships in the system dynamics and naturally scale to problems with high dimensionality and discrete or continuous variables. In addition, neural network parameters can be dynamically updated efficiently based on data availability. DBMMs can thus be used to infer a belief variable, thus enabling the derivation of POMDP solutions over the belief space. We evaluate the efficacy of the proposed methodology by evaluating the capability of model-formulation agnostic inference of DBMMs in benchmark problems that include discrete and continuous variables.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。