深度达1024层的自监督强化学习模型显著提升目标达成能力
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
- 用超深网络（最多1024层）实现无奖励、无示范的自监督目标导向学习
- 在模拟任务中性能提升2到50倍，成功率达基准的数倍
- 适合研究深度强化学习可扩展性的学者和追求高鲁棒性智能体的工程师
自监督学习在语言和视觉领域已取得突破，但在强化学习（RL）领域进展有限。本文研究了自监督强化学习的关键构建模块，发现网络深度是提升可扩展性的关键因素。以往多数论文使用2-5层浅层结构，而本文将网络深度扩展至1024层，显著提升性能。实验在无监督目标条件设置下进行，不提供演示或奖励信号，智能体需从零开始探索并最大化到达指定目标的概率。在模拟的运动与操作任务上，该方法使自监督对比强化学习算法性能提升2至50倍，优于其他目标条件基线。增加模型深度不仅提高成功率，还改变了所学行为的质态。项目主页与代码见：https://wang-kevin3290.github.io/scaling-crl/
原文摘要 · Abstract (English)
Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ - $50\times$, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。