用微分方程约束隐状态演化，让强化学习更贴合环境真实动态。
Flowing Through States: Neural ODE Regularization for Reinforcement Learning
- 将马尔可夫决策过程类比为常微分方程流，显式建模隐状态演化
- 在A2C和PPO算法中提升在Atari与网格世界任务上的性能
- 适用于多种深度学习代理，尤其增强强化学习的表征一致性
应用于序列决策任务的神经网络通常依赖于环境状态的隐表示。尽管环境动态决定了语义状态的演变，但对应的隐状态转移通常未被显式建模，导致两者之间存在潜在错位。本文通过将马尔可夫决策过程（MDP）轨迹与常微分方程（ODE）流类比，提出一种基于神经ODE的正则化方法：显式建模隐状态演化，使隐表示遵循一致的ODE流，从而对齐表征学习与环境动态。该方法广泛适用于深度学习智能体，我们将其集成至Actor-Critic算法，在多个标准Atari基准和PPO的网格世界环境中均取得显著性能提升。
原文摘要 · Abstract (English)
Neural networks applied to sequential decision-making tasks typically rely on latent representations of environment states. While environment dynamics dictate how semantic states evolve, the corresponding latent transitions are usually left implicit, creating a potential misalignment between the two. We propose to model latent dynamics explicitly by drawing an analogy between Markov decision process (MDP) trajectories and ordinary differential equation (ODE) flows: in both cases, the current state fully determines its successors. Building on this view, we introduce a neural ODE-based regularization method that enforces latent embeddings to follow consistent ODE flows, thereby aligning representation learning with environment dynamics. Although broadly applicable to deep learning agents, we demonstrate its effectiveness in reinforcement learning by integrating it into Actor-Critic algorithms. Our approach yields major performance gains across various standard Atari benchmarks for A2C and gridworld environments for PPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。