提出后收敛期最优策略迁移方法,解决仿真到现实部署的盲目挑选问题。
Post-Convergence Sim-to-Real Policy Transfer: A Principled Alternative to Cherry-Picking
- 基于最坏情况性能优化,构建凸二次约束线性规划模型。
- 在真实实验室测试中显著提升仿真训练策略的迁移成功率。
- 适合关注仿真到现实稳定迁移的机器人控制研究者。
基于学习的方法,尤其是强化学习(RL),广泛用于开发自主代理(如四足机器人)的控制策略。RL训练通过迭代优化策略来最大化预定义奖励(或最小化代价/损失),通常从随机初始化策略开始,经验期望奖励呈现整体上升趋势。尽管某些策略会暂时陷入局部最优,但良好的训练过程一般会收敛至具有噪声波动的奖励水平。然而,真实世界部署的策略选择很少是分析性的(即简单选取奖励最高的策略),而是常通过试错完成。现有研究大多聚焦于收敛前阶段,采用领域随机化、多保真度训练、对抗训练和架构创新等技术。这些方法无法消除奖励不可避免的收敛轨迹和噪声波动,导致策略选择仍依赖启发式或随意挑选。本文针对后收敛期的仿真到现实迁移问题,提出一种最坏情况性能迁移优化方法,将其建模为凸二次约束线性规划问题。大量实验表明,该方法能有效将基于RL的步态策略从仿真成功迁移至真实实验室测试。
原文摘要 · Abstract (English)
Learning-based approaches, particularly reinforcement learning (RL), have become widely used for developing control policies for autonomous agents, such as locomotion policies for legged robots. RL training typically maximizes a predefined reward (or minimizes a corresponding cost/loss) by iteratively optimizing policies within a simulator. Starting from a randomly initialized policy, the empirical expected reward follows a trajectory with an overall increasing trend. While some policies become temporarily stuck in local optima, a well-defined training process generally converges to a reward level with noisy oscillations. However, selecting a policy for real-world deployment is rarely an analytical decision (i.e., simply choosing the one with the highest reward) and is instead often performed through trial and error. To improve sim-to-real transfer, most research focuses on the pre-convergence stage, employing techniques such as domain randomization, multi-fidelity training, adversarial training, and architectural innovations. However, these methods do not eliminate the inevitable convergence trajectory and noisy oscillations of rewards, leading to heuristic policy selection or cherry-picking. This paper addresses the post-convergence sim-to-real transfer problem by introducing a worst-case performance transference optimization approach, formulated as a convex quadratic-constrained linear programming problem. Extensive experiments demonstrate its effectiveness in transferring RL-based locomotion policies from simulation to real-world laboratory tests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。