用价值网络当探索器，零新增参数实现高效稳定探索
CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
- 用强化学习中的价值网络驱动探索，不加新参数
- 理论保证亚线性后悔，实验在多个环境表现优异
- 代码仅需约10行，适合追求简洁高效的从业者
探索仍是深度强化学习中的核心挑战，现有方法或缺乏理论保证，或实际效果不佳。本文提出CAE（Critic as an Explorer），一种轻量级方法，将标准深度强化学习算法中的价值网络重新用于驱动探索，无需引入额外参数。CAE结合多臂老虎机技术与定制缩放策略，实现高效探索，具备可证明的亚线性后悔界和强经验稳定性。其代码仅需约10行。针对价值网络难以学习的复杂任务，提出CAE+——在保持实现简便的前提下，引入辅助网络，参数增加不足1%，额外代码约10行。在MuJoCo、MiniHack和Habitat上的大量实验验证了CAE与CAE+的有效性，展现了理论严谨性与实践效率的统一。
原文摘要 · Abstract (English)
Exploration remains a fundamental challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short in practical effectiveness. In this paper, we propose CAE, i.e., the Critic as an Explorer, a lightweight approach that repurposes the value networks in standard deep RL algorithms to drive exploration, without introducing additional parameters. CAE leverages multi-armed bandit techniques combined with a tailored scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and strong empirical stability. Remarkably, it is simple to implement, requiring only about 10 lines of code. For complex tasks where learning reliable value networks is difficult, we introduce CAE+, an extension of CAE that incorporates an auxiliary network. CAE+ increases the parameter count by less than 1% while preserving implementation simplicity, adding roughly 10 additional lines of code. Extensive experiments on MuJoCo, MiniHack, and Habitat validate the effectiveness of CAE and CAE+, highlighting their ability to unify theoretical rigor with practical efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。