用动态规划提取状态特征，构建无需外部模型的环境建模框架。
DeeP-Mod: Deep Dynamic Programming based Environment Modelling using Feature Extraction
- 通过动态规划训练网络，确保输出为纯状态值而非状态-动作混合表示。
- 提取的特征可复用，简化模型后收敛更快且抗噪声能力更强。
- 适合在无先验环境模型条件下快速部署强化学习系统。
DeeP-Mod 框架利用深度动态规划网络（DDPN）提取的特征构建环境模型，该 DDPN 通过深度 Q 网络（DQN）训练。传统 DQN 在深层中因状态-动作表示混合导致状态信息丢失。为此，本文采用动态规划（DP）训练 DDPN，确保输出为状态值而非状态-动作对。从 DDPN 提取的特征保留了完整状态信息，实现任务与动作集独立。我们证明，基于原始 DDPN 提取的特征，可训练一个简化版 DDPN，其在噪声环境下收敛更快，性能优于原模型。最终提出 DeeP-Mod 框架，通过分析特征随动作演化的规律构建环境模型。第二个 DDPN 直接从该特征模型学习，而非原始状态，从而获得有效的特征-价值表示和最优策略。关键优势在于整个过程无需外部环境模型，使 DDPN 可适用于多种环境。
原文摘要 · Abstract (English)
The DeeP-Mod framework builds an environment model using features from a Deep Dynamic Programming Network (DDPN), trained via a Deep Q-Network (DQN). While Deep Q-Learning is effective in decision-making, state information is lost in deeper DQN layers due to mixed state-action representations. We address this by using Dynamic Programming (DP) to train a DDPN, where Value Iteration ensures the output represents state values, not state-action pairs. Extracting features from the DDPN preserves state information, enabling task and action set independence. We show that a reduced DDPN can be trained using features extracted from the original DDPN trained on an identical problem. This reduced DDPN achieves faster convergence under noise and outperforms the original DDPN. Finally, we introduce the DeeP-Mod framework, which creates an environment model using the evolution of features extracted from a DDPN in response to actions. A second DDPN, which learns directly from this feature model rather than raw states, can learn an effective feature-value representation and thus optimal policy. A key advantage of DeeP-Mod is that an externally defined environment model is not needed at any stage, making DDPN applicable to a wide range of environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。