从价值函数推断环境动态，打通了无模型与有模型强化学习的壁垒。
Inferring Transition Dynamics from Value Functions
- 通过重排贝尔曼方程，揭示价值函数隐含环境转移动态
- 直接由收敛的价值函数推导出动态模型，无需额外训练
- 为模型无关与模型相关RL融合提供理论基础，适合研究者参考
在强化学习中，价值函数通常用于求解贝尔曼方程，该方程将当前值与未来值关联。这种时间依赖性暗示价值函数可能包含环境转移动态的隐式信息。通过重新排列贝尔曼方程，我们证明收敛的价值函数编码了环境底层动态模型。基于此洞察，我们提出一种直接从价值函数推断动态模型的简单方法，可能减少对显式模型学习的需求。此外，我们探讨了下一状态可识别性的挑战，分析了推断动态模型有良好定义的条件。本工作为利用价值函数进行动态建模提供了理论基础，并开辟了连接无模型与有模型强化学习的新路径。
原文摘要 · Abstract (English)
In reinforcement learning, the value function is typically trained to solve the Bellman equation, which connects the current value to future values. This temporal dependency hints that the value function may contain implicit information about the environment's transition dynamics. By rearranging the Bellman equation, we show that a converged value function encodes a model of the underlying dynamics of the environment. We build on this insight to propose a simple method for inferring dynamics models directly from the value function, potentially mitigating the need for explicit model learning. Furthermore, we explore the challenges of next-state identifiability, discussing conditions under which the inferred dynamics model is well-defined. Our work provides a theoretical foundation for leveraging value functions in dynamics modeling and opens a new avenue for bridging model-free and model-based reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。