最大熵强化学习在复杂控制中可能因过度探索而失效
When Maximum Entropy Misleads Policy Optimization
- 用熵最大化增强探索，但可能导致策略偏离最优
- 实验显示在需精确控制的任务中，最大熵方法会失败
- 适合关注策略稳定性和奖励设计的研究者
最大熵强化学习（MaxEnt RL）是实现高效学习和鲁棒性能的主流方法，但在实际中，面对对性能要求高的控制任务时，其表现却不如非最大熵算法。本文分析了鲁棒性与最优性之间的权衡如何影响最大熵算法在复杂控制任务中的表现：虽然熵最大化能提升探索能力和鲁棒性，但也可能误导策略优化，导致在需要精确、低熵策略的任务中失败。通过在多种控制问题上的实验，我们具体展示了这种误导效应。该分析有助于更好地理解如何在挑战性控制任务中平衡奖励设计与熵最大化。
原文摘要 · Abstract (English)
The Maximum Entropy Reinforcement Learning (MaxEnt RL) framework is a leading approach for achieving efficient learning and robust performance across many RL tasks. However, MaxEnt methods have also been shown to struggle with performance-critical control problems in practice, where non-MaxEnt algorithms can successfully learn. In this work, we analyze how the trade-off between robustness and optimality affects the performance of MaxEnt algorithms in complex control tasks: while entropy maximization enhances exploration and robustness, it can also mislead policy optimization, leading to failure in tasks that require precise, low-entropy policies. Through experiments on a variety of control problems, we concretely demonstrate this misleading effect. Our analysis leads to better understanding of how to balance reward design and entropy maximization in challenging control problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。