用柯尔莫哥洛夫算子思想让强化学习更稳定,性能提升60%。
KIPPO: Koopman-Inspired Proximal Policy Optimization
- 引入柯尔莫哥洛夫算子思想,构建线性隐空间表示系统动态
- 在连续控制任务中性能提升6-60%,方差降低最高达91%
- 可无缝接入现有PPO框架,无需改动核心网络结构
强化学习在多个领域取得显著进展,基于梯度的策略优化方法如近端策略优化(PPO)因其性能、训练稳定性与计算效率的平衡而广受欢迎。然而,在复杂非线性动力学环境中设计有效控制策略仍具挑战,梯度估计高方差与非凸优化景观常导致学习轨迹不稳定。柯尔莫哥洛夫算子理论提供了一种通过无限维线性算子作用于测量函数高维空间来研究非线性系统的强大框架,相比非线性系统,线性系统更简单、可预测且易于分析。本文提出柯尔莫哥洛夫启发的近端策略优化(KIPPO),在保留策略学习关键特征的同时,学习系统动态的近似线性隐空间表示。该方法通过一个柯尔莫哥洛夫近似辅助网络实现,可添加至基础策略优化算法中,无需改变核心策略或价值函数架构。大量实验表明,与基线PPO相比,KIPPO在多种连续控制任务中实现6-60%的性能提升,同时将变异性降低高达91%。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has made significant strides in various domains, and policy gradient methods like Proximal Policy Optimization (PPO) have gained popularity due to their balance in performance, training stability, and computational efficiency. These methods directly optimize policies through gradient-based updates. However, developing effective control policies for environments with complex and non-linear dynamics remains a challenge. High variance in gradient estimates and non-convex optimization landscapes often lead to unstable learning trajectories. Koopman Operator Theory has emerged as a powerful framework for studying non-linear systems through an infinite-dimensional linear operator that acts on a higher-dimensional space of measurement functions. In contrast with their non-linear counterparts, linear systems are simpler, more predictable, and easier to analyze. In this paper, we present Koopman-Inspired Proximal Policy Optimization (KIPPO), which learns an approximately linear latent-space representation of the underlying system's dynamics while retaining essential features for effective policy learning. This is achieved through a Koopman-approximation auxiliary network that can be added to the baseline policy optimization algorithms without altering the architecture of the core policy or value function. Extensive experimental results demonstrate consistent improvements over the PPO baseline with 6-60% increased performance while reducing variability by up to 91% when evaluated on various continuous control tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。