通过偏差采样与GPU加速，让强化学习控制更高效可扩展。
CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization
- 用价值函数特性引导初始状态采样，提升数据效率
- 结合GPU加速轨迹优化，计算速度显著提升
- 在四足机器人上实现实时控制，适合高维系统
轨迹优化（TO）与强化学习（RL）在求解最优控制问题时各有优势：TO能高效计算局部最优解，但对非凸问题易失效；而RL虽对非凸性更鲁棒，但计算成本高昂。CACTO通过学习一个热启动策略，引导TO求解器找到低代价轨迹，融合二者优势。然而其可扩展性受限，系统复杂度上升时TO计算成本急剧增加。本文提出CACTO-BIC，通过利用局部最优策略对应的价值函数特性，对初始状态采样进行偏差调整，提升数据效率；同时借助GPU加速轨迹优化过程，显著降低计算时间。实验表明，相比CACTO，本方法在样本效率和计算速度上均有提升；与PPO对比，可在更短时间内获得相近性能。在AlienGO四足机器人上的实验显示，CACTO-BIC可成功拓展至高维系统，适用于实时控制场景。
原文摘要 · Abstract (English)
Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。