提出三种世界模型:环境、智能体与联合系统,揭示其预测结构本质。
World models of environment, agent and joint agent-environment systems

- 基于信息通道区分三类世界模型:环境、智能体与联合系统
- 支持受限模型可将无限状态简化为有限状态,提升可计算性
- 适用于强化学习中复杂系统的建模与分析,尤其适合研究闭环交互
世界模型是基于模型的强化学习核心。传统讨论聚焦于预测变量(如观测、奖励、状态等),我们提出更根本的区分:建模的是哪个信道。考虑三种情形:环境信道 $O_{:} ackslash A_{:}$、智能体信道 $A_{:} ackslash O_{:}$,以及联合过程 $(A, O)_{:}$(无输入)。利用计算力学,定义这三类情况的典型预测模型——$ε$-transducer 或 $ε$-machine。典型环境模型恢复标准的预测状态表示,其余两类则分别给出智能体与联合系统的类似范式。进一步构建由闭环耦合诱导的支持受限模型,其预测等价性覆盖实际交互产生的延续序列。关键结构结果表明:支持受限环境状态通过联合因果状态因子化,且其转移结构直接来自联合模型;智能体侧构造具对偶性。最后以一个部分可观测马尔可夫决策过程(POMDP)/控制器实例说明:无约束环境模型有无穷多状态,而由耦合诱导的支持受限模型却为有限。该框架明确了不同世界模型所建模的对象,并揭示了耦合与支持限制如何改变其典型预测结构与复杂度。
原文摘要 · Abstract (English)
World models are a central component of model-based reinforcement learning. They are usually discussed in terms of what variables they predict, such as observations, rewards, states, latent or information states. We argue that there is a prior distinction: which channel they model. We consider three cases: the environment channel $O_{:} \mid A_{:}$, the agent channel $A_{:} \mid O_{:}$, and the realised joint process $(A, O)_{:}$, equivalently viewed as a channel with no inputs. Using computational mechanics, we define canonical predictive models for these three cases as $ε$-transducers or $ε$-machines. Canonical environment models recover standard predictive state representations, while the other two give analogous notions of canonical models for the agent and the joint system. We then build canonical support-restricted environment and agent models induced by closed-loop coupling, whose predictive equivalences range over continuations supported by the realised interaction. The key structural result is that canonical support-restricted environment states factor through the canonical joint causal states, and their transition structure is induced directly from the joint model; the agent-side construction is dual. Finally, we give a POMDP/controller example in which the unrestricted environment model has infinitely many states while the canonical support-restricted model induced by the coupling is finite. The framework clarifies what different world models are models of, and how coupling and support restriction can change their canonical predictive structure and complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。