arXiv:2608.02519cs.LGcs.AI2026-08

提出无需限制条件的分布感知规划方法,提升强化学习在不确定性环境中的稳定性。

Analytic Planning under Uncertainty with Moment Closure

  • 用二次动作价值参数化将贝尔曼更新简化为仅依赖值函数期望
  • 通过高斯转移与径向基值函数匹配,实现均值和协方差的闭式传播
  • 在连续控制中降低目标方差,校准预测不确定性,适合学习分布模型的场景

在随机环境中,基于模型的强化学习需要考虑预测不确定性。传统上,解析传播完整状态分布需对策略或奖励结构施加严格限制,导致现代深度强化学习主要采用引入显著目标方差的随机采样,或完全忽略预测协方差的确定性点估计。本文研究是否可在无这些限制的情况下实现分布感知规划。通过二次动作价值参数化,我们将贝尔曼更新简化为仅关于状态值函数的期望;核心思想是预测转移分布与值函数类之间的兼容性原则,使得该期望可解析地表示为分布矩的函数。我们以高斯转移模型搭配径向基值函数实例化此原则,获得闭式备份,可同时传播预测均值与协方差。实验表明,该方法在连续控制任务中显著降低目标方差,并在随机观测下提供校准良好的预测不确定性,为学习分布模型的规划提供了原则性框架。

原文摘要 · Abstract (English)

Effective model-based reinforcement learning in stochastic environments requires planning that accounts for predictive uncertainty. Propagating full state distributions analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable. Consequently, modern deep reinforcement learning has largely retreated to either stochastic sampling, which introduces significant target variance, or deterministic point estimates that ignore predictive covariance entirely. We investigate whether distribution-aware planning is possible without these constraints. Using a quadratic action-value parameterization, we first reduce the Bellman backup to an expectation over the state-value function alone; the key idea is then a compatibility principle between the predictive transition distribution and the value function class, under which this expectation is analytic in the distribution's moments. We instantiate this principle with a Gaussian transition model paired with a radial-basis value function, yielding a closed-form backup that propagates both predictive mean and covariance. Empirically, our approach reduces target variance and yields well-calibrated predictive uncertainty under stochastic observations in continuous control, providing a principled framework for planning with learned distribution models.

强化学习不确定性建模分布规划连续控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。