让智能体主动改写环境模型，从而提升长期奖励
Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions
- 上层用可配置动作修改下层环境动态模型
- 双层优化：同时学习配置策略和基础动作策略
- 适合想突破固定环境限制的强化学习研究者
强化学习通常假设环境给定且固定，智能体只能被动适应。本文提出多层可配置时变马尔可夫决策过程（MCTVMDP），允许智能体通过上层的模型更改动作，主动重构底层环境的转移机制。这种对世界动态模型的重新配置可能带来更高回报。目标是联合优化上层的配置策略和下层的基础动作策略，以最大化期望长期回报。
原文摘要 · Abstract (English)
Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。