用可微模拟加速视觉策略学习，训练快4倍且更稳定。
Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
- 分离渲染与计算图，降低资源开销
- 复杂任务上最终回报提升4倍，单卡4小时完成跑步训练
- 适合需要快速迭代的视觉控制研究者
本文提出一种高效视觉策略学习算法，利用可微模拟和一阶解析策略梯度。通过将渲染过程从计算图中解耦，无需专用可微渲染软件即可无缝集成现有可微模拟生态。该设计不仅降低计算与内存开销，还有效抑制策略梯度范数，实现更稳定平滑的优化。在现代GPU加速仿真环境下，我们在标准视觉控制基准上评估该方法，实验表明其显著缩短了实际训练时间，并在最终回报上全面超越所有基线方法。尤其在人形机器人行走等复杂任务中，最终回报提升4倍，且仅用单张GPU在4小时内成功训练出跑步策略。
原文摘要 · Abstract (English)
In this work, we propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients. Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software. This decoupling not only reduces computational and memory overhead but also effectively attenuates the policy gradient norm, leading to more stable and smoother optimization. We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns. Notably, on complex tasks such as humanoid locomotion, our method achieves a $4\times$ improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。