Squint让视觉强化学习训练更快，15分钟内完成8个复杂任务，且可真实机器人部署。
Squint: Fast Visual Reinforcement Learning for Sim-to-Real Robotics
- 结合并行仿真与分布批评器，用分辨率压缩减少计算开销。
- 在单张RTX 3090上15分钟内训练完成，多数任务6分钟内收敛。
- 适用于需快速从仿真迁移到真实机器人的视觉控制任务。
视觉强化学习在机器人领域极具吸引力，但成本高昂——离策略方法虽样本高效却速度慢；在策略方法可良好并行但浪费样本。近期研究表明，在基于状态的控制中，离策略方法比在策略方法更节省实际运行时间。将此拓展至视觉仍具挑战性，因高维图像输入使训练动态复杂化，并带来显著存储和编码开销。为此，我们提出Squint，一种视觉软演员-评论家方法，其实际运行时间快于先前的视觉离策略和在策略方法。Squint通过并行仿真、分布批评器、分辨率压缩、层归一化、优化的更新/数据比例及高效实现达成加速。我们在ManiSkill3中引入的新SO-101任务集（含8个强域随机化的操作任务）上进行评估，并展示了向真实SO-101机器人的仿真到现实迁移能力。仅用单张RTX 3090 GPU训练15分钟，多数任务在6分钟内完成收敛。
原文摘要 · Abstract (English)
Visual reinforcement learning is appealing for robotics but expensive -- off-policy methods are sample-efficient yet slow; on-policy methods parallelize well but waste samples. Recent work has shown that off-policy methods can train faster than on-policy methods in wall-clock time for state-based control. Extending this to vision remains challenging, where high-dimensional input images complicate training dynamics and introduce substantial storage and encoding overhead. To address these challenges, we introduce Squint, a visual Soft Actor Critic method that achieves faster wall-clock training than prior visual off-policy and on-policy methods. Squint achieves this via parallel simulation, a distributional critic, resolution squinting, layer normalization, a tuned update-to-data ratio, and an optimized implementation. We evaluate on the SO-101 Task Set, a new suite of eight manipulation tasks in ManiSkill3 with heavy domain randomization, and demonstrate sim-to-real transfer to a real SO-101 robot. We train policies for 15 minutes on a single RTX 3090 GPU, with most tasks converging in under 6 minutes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。