重新审视互信息技能学习，发现其可解释METRA的优异表现
Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning
- 用对比后继特征重构互信息技能学习框架
- 新方法性能媲美METRA但结构更简单
- 适合研究强化学习表征与技能学习的学者
自监督学习有望解决强化学习中的探索、表征学习和奖励设计等关键挑战。近期工作METRA提出摒弃互信息而优化特定Wasserstein距离对性能提升至关重要。本文认为，METRA的收益很大程度上可用现有互信息技能学习（MISL）框架解释。我们提出一种新的MISL方法——对比后继特征，该方法在保持METRA优异性能的同时减少组件复杂度，并揭示了技能学习、对比表征学习与后继特征之间的联系。通过细致的消融实验，进一步阐明了本方法及METRA的关键构成要素。
原文摘要 · Abstract (English)
Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。