用潜在动态统一无模型效率与模型表示,无需规划开销。
Unifying Model-Free Efficiency and Model-Based Representations via Latent Dynamics
- 将状态动作对嵌入价值函数近似线性的潜在空间
- 80个环境上性能媲美或超越专用算法,参数量少得多
- 适合追求跨域泛化且资源受限的强化学习应用
我们提出统一潜在动态(ULD),一种新型强化学习算法,将无模型方法的高效性与模型基础方法的表征优势相结合,且不引入规划开销。通过将状态-动作对嵌入到一个真值函数近似线性的潜在空间中,该方法在多种任务间仅需一组超参数——从低维连续控制、像素输入到高维Atari游戏均适用。我们在温和条件下证明,基于嵌入的时序差分更新的不动点与对应线性模型基础价值扩展一致,并推导出嵌入保真度与价值近似质量之间的显式误差界。实际中,ULD采用编码器、价值和策略网络的同步更新,辅以短程预测动态的辅助损失和奖励尺度归一化,确保稀疏奖励下稳定学习。在涵盖Gym运动、DeepMind Control(本体感受与视觉)及Atari的80个环境上评估,其表现匹配或超越专用无模型与通用模型基线,实现跨域能力仅需极少调参且参数量仅为后者的几分之一。结果表明,仅靠对齐价值的潜在表征即可实现传统全模型规划所赋予的适应性与样本效率。
原文摘要 · Abstract (English)
We present Unified Latent Dynamics (ULD), a novel reinforcement learning algorithm that unifies the efficiency of model-free methods with the representational strengths of model-based approaches, without incurring planning overhead. By embedding state-action pairs into a latent space in which the true value function is approximately linear, our method supports a single set of hyperparameters across diverse domains -- from continuous control with low-dimensional and pixel inputs to high-dimensional Atari games. We prove that, under mild conditions, the fixed point of our embedding-based temporal-difference updates coincides with that of a corresponding linear model-based value expansion, and we derive explicit error bounds relating embedding fidelity to value approximation quality. In practice, ULD employs synchronized updates of encoder, value, and policy networks, auxiliary losses for short-horizon predictive dynamics, and reward-scale normalization to ensure stable learning under sparse rewards. Evaluated on 80 environments spanning Gym locomotion, DeepMind Control (proprioceptive and visual), and Atari, our approach matches or exceeds the performance of specialized model-free and general model-based baselines -- achieving cross-domain competence with minimal tuning and a fraction of the parameter footprint. These results indicate that value-aligned latent representations alone can deliver the adaptability and sample efficiency traditionally attributed to full model-based planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。