小演员搭配乐观批评者，能改善强化学习性能
Optimistic critics can empower small actors
- 采用不对称架构，让演员网络更小，批评者更复杂
- 小演员导致价值低估，引发过拟合批评者
- 通过乐观批评者缓解数据收集问题，适合研究高效模型
Actor-critic 方法在深度强化学习中占据核心地位。主流做法是使用对称架构，即演员和批评者具有相同的网络结构和参数量。然而，近期研究主张采用非对称设置，尤其是更小的演员网络。本文通过广泛实验与分析发现，通常情况下，较小的演员会导致性能下降和批评者过拟合。分析表明，价值低估引起的不良数据收集是主要原因，并强调了批评者在缓解该问题中的关键作用。本文探索了减轻价值低估的技术，为不对称 actor-critic 方法的进一步研究提供了可能。
原文摘要 · Abstract (English)
Actor-critic methods have been central to many of the recent advances in deep reinforcement learning. The most common approach is to use symmetric architectures, whereby both actor and critic have the same network topology and number of parameters. However, recent works have argued for the advantages of asymmetric setups, specifically with the use of smaller actors. We perform broad empirical investigations and analyses to better understand the implications of this and find that, in general, smaller actors result in performance degradation and overfit critics. Our analyses suggest poor data collection, due to value underestimation, as one of the main causes for this behavior, and further highlight the crucial role the critic can play in alleviating this pathology. We explore techniques to mitigate the observed value underestimation, which enables further research in asymmetric actor-critic methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。