让并行智能体各走各路，提升探索多样性，加速强化学习训练。
Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story
- 通过最大化状态熵，让并行智能体策略差异化，减少重复探索。
- 实验显示比相同智能体方案更优，且能与批量强化学习方法协同增效。
- 理论分析证明专用采样分布收敛更快，适合大规模强化学习场景。
并行数据收集重新定义了强化学习，显著提升了效率，并推动了大规模真实世界应用的突破。在此范式下，N个相同的智能体在环境模拟器的N个副本中运行，使数据收集速度提升N倍。一个关键问题是：是否通过让并行智能体的策略差异化，可以突破N倍加速的上限？本文提出一种新型学习框架，旨在最大化并行设置下的数据熵。该方法精心平衡单个智能体的熵与跨智能体间的多样性，有效降低冗余。这一思想通过集中式策略梯度方法实现，在与相同智能体系统对比的实证评估中表现出色，并可与批量强化学习技术协同利用数据多样性。最后，我们提供了一项原创的浓度分析，表明专用并行采样分布具有更快的收敛速率，支持了本方法的有效性，也可能具有独立研究价值。
原文摘要 · Abstract (English)
Parallel data collection has redefined Reinforcement Learning (RL), unlocking unprecedented efficiency and powering breakthroughs in large-scale real-world applications. In this paradigm, $N$ identical agents operate in $N$ replicas of an environment simulator, accelerating data collection by a factor of $N$. A critical question arises: \textit{Does specializing the policies of the parallel agents hold the key to surpass the $N$ factor acceleration?} In this paper, we introduce a novel learning framework that maximizes the entropy of collected data in a parallel setting. Our approach carefully balances the entropy of individual agents with inter-agent diversity, effectively minimizing redundancies. The latter idea is implemented with a centralized policy gradient method, which shows promise when evaluated empirically against systems of identical agents, as well as synergy with batch RL techniques that can exploit data diversity. Finally, we provide an original concentration analysis that shows faster rates for specialized parallel sampling distributions, which supports our methodology and may be of independent interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。