如何在复杂模拟器中选择最佳状态近似方法,让重置更准确。
Selecting Belief-State Approximations in Simulators with Latent States
- 提出基于采样的信念状态选择新算法,无需模型内部信息。
- 发现观察导向选择在特定重置策略下会失效,需谨慎设计下游方法。
- 揭示了模拟器重置中的理论复杂性,适合强化学习与系统仿真研究者。
状态重置是模拟器的基础能力,支持基于样本的规划并实现真实数据校准。但在包含潜在变量的复杂模拟器中,重置需从观测历史推断潜在状态的后验分布(即信念状态),而精确采样通常不可行。本文将该问题转化为仅依赖采样访问的通用条件分布选择任务,提出新算法与分析。我们发现信念状态选择可有两种形式:基于潜在状态的直接选择,以及基于观测的间接选择。两者在不同回放策略下的表现差异显著——观察导向选择在最自然的单次重置策略下可能失败,但在重复重置策略下仍能保证效果。结合分布偏移和采样策略选择等讨论,本文揭示了此看似简单问题背后的丰富算法选择与理论挑战。
原文摘要 · Abstract (English)
State resetting is a fundamental but often overlooked capability of simulators. It supports sample-based planning by allowing resets to previously encountered simulation states, and enables calibration of simulators using real data by resetting to states observed in real-system traces. While often taken for granted, state resetting in complex simulators can be nontrivial: when the simulator comes with latent variables (states), state resetting requires sampling from the posterior over the latent state given the observable history, a.k.a. the belief state (Silver and Veness, 2010). While exact sampling is often infeasible, many approximate belief-state samplers can be constructed, raising the question of how to select among them using only sampling access to the simulator. In this paper, we show that this problem reduces to a general conditional distribution-selection task and develop a new algorithm and analysis under sampling-only access. Building on this reduction, the belief-state selection problem admits two different formulations: latent state-based selection, which directly targets the conditional distribution of the latent state, and observation-based selection, which targets the induced distribution over the observation. Interestingly, these formulations differ in how their guarantees interact with the downstream roll-out methods: perhaps surprisingly, observation-based selection may fail under the most natural roll-out method (which we call Single-Reset) but enjoys guarantees under the less conventional alternative (which we call Repeated-Reset). Together with discussion on issues such as distribution shift and the choice of sampling policies, our paper reveals a rich landscape of algorithmic choices, theoretical nuances, and open questions, in this seemingly simple problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。