发现潜空间模型的不确定性估计会因吸引子效应失效,导致高估奖励。
Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models

- 用集成分歧衡量潜空间模型的信念不确定度
- 长期滚动推演中误差累积却因吸引子效应被低估
- 适合关注强化学习中模型信任度与安全性的研究者
基于模型的强化学习区分了基于本体感受状态的动力学模型与通常作用于高维图像观测的潜空间动力学模型。其中,Dreamer的递归状态空间模型(RSSM)已成为主流架构。尽管集成方法在本体感受动力学中能有效缓解模型滥用、引导探索或促进谨慎决策,其在潜空间动力学中的表现仍不明确。实验表明,虽然集成分歧可捕捉局部信念不确定度,但无法可靠反映长期潜空间滚动推演中累积的全局模型误差。我们发现存在吸引子行为,使推演轨迹被拉向支持度高的潜空间区域,即使与真实环境动态偏差增大,不确定性仍下降。当这些吸引子区域对应高奖励行为时,模型可能过度高估回报。结果揭示了RSSM中信念不确定度估计的结构性局限,挑战了信念不确定度从本体感受到潜空间可直接迁移的假设。
原文摘要 · Abstract (English)
Model-based reinforcement learning distinguishes between dynamics models operating on proprioceptive states and latent dynamics models typically operating on high-dimensional image observations. Among the latter, Dreamer's Recurrent State Space Model (RSSM) has emerged as a dominant architecture. While ensemble-based epistemic uncertainty has proven effective in proprioceptive dynamics for mitigating model exploitation, guiding exploration, or promoting caution, its behavior in latent dynamics remains largely unexplored. Our experiments reveal that although ensemble disagreement captures local epistemic uncertainty, it does not reliably reflect global compounding model error accumulated over prolonged RSSM latent rollouts. We provide evidence for an attractor behavior that draws rollouts toward well-supported latent regions, where uncertainty diminishes despite increasing discrepancies from the true environment dynamics. This can cause the model to overestimate returns when attractor regions correspond to high-reward behaviors. Our findings reveal a structural limitation of epistemic uncertainty estimation in RSSMs and challenge the assumption that epistemic uncertainty estimation transfers directly from proprioceptive to latent dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。