用退火重要性重采样改进在线决策，解决观测噪声下的粒子退化问题。
Observation Adaptation via Annealed Importance Resampling for Partially Observable Markov Decision Processes
- 构建一系列桥接分布，逐步逼近最优后验分布
- 在多个挑战性任务中显著优于现有方法
- 适合需要实时适应高信息量观测的强化学习场景
部分可观测马尔可夫决策过程（POMDP）是随机环境中状态不确定下序列决策的通用数学模型。通常采用在线求解器，能实时适应新信息。现有在线算法多使用基于重要性重采样的粒子滤波器更新信念分布。由于直接从最新观测和先前状态推断理想状态分布不可行，粒子滤波器通过预测与重采样步骤传播状态并调整权重，近似后验信念分布。然而，当状态转移模型与后验分布偏差较大时，尤其在接收高信息量观测时，重要性重采样常导致粒子退化与样本贫化。本文提出一种新方法，通过迭代蒙特卡洛步骤构建状态转移分布与最优分布之间的系列桥接分布，更好地处理在线POMDP求解器中的噪声观测。实验表明，该算法在多个具有挑战性的POMDP领域中显著优于当前最优方法。
原文摘要 · Abstract (English)
Partially observable Markov decision processes (POMDPs) are a general mathematical model for sequential decision-making in stochastic environments under state uncertainty. POMDPs are often solved \textit{online}, which enables the algorithm to adapt to new information in real time. Online solvers typically use bootstrap particle filters based on importance resampling for updating the belief distribution. Since directly sampling from the ideal state distribution given the latest observation and previous state is infeasible, particle filters approximate the posterior belief distribution by propagating states and adjusting weights through prediction and resampling steps. However, in practice, the importance resampling technique often leads to particle degeneracy and sample impoverishment when the state transition model poorly aligns with the posterior belief distribution, especially when the received observation is highly informative. We propose an approach that constructs a sequence of bridge distributions between the state-transition and optimal distributions through iterative Monte Carlo steps, better accommodating noisy observations in online POMDP solvers. Our algorithm demonstrates significantly superior performance compared to state-of-the-art methods when evaluated across multiple challenging POMDP domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。