让智能体学会适时放弃，避免无效尝试提升训练效率
The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
- 基于Q值与梯度统计设计早停机制，判断何时终止无意义回合
- 在MuJoCo和DeepMind Control Suite上显著减少无效采样
- 适用于各类离策略强化学习算法，提升样本利用效率
离策略深度强化学习通常使用回放缓冲区复用历史经验以提高样本效率；但当数据信息量低且偏离学习目标时，会污染缓冲区，加剧优化困难并浪费环境交互。本文指出，这种现象源于‘沉没成本谬误’——倾向于将回合进行到底。为此提出轻量级的LEAST机制，通过分析Q值与梯度统计，使智能体能主动识别并提前终止无产出的回合。实验表明，该方法在多种RL算法上均提升了学习效率，验证于MuJoCo与DeepMind Control Suite基准测试。
原文摘要 · Abstract (English)
Off-policy deep reinforcement learning (RL) typically leverages replay buffers for reusing past experiences during learning. This can help improve sample efficiency when the collected data is informative and aligned with the learning objectives; when that is not the case, it can have the effect of "polluting" the replay buffer with data which can exacerbate optimization challenges in addition to wasting environment interactions due to wasteful sampling. We argue that sampling these uninformative and wasteful transitions can be avoided by addressing the sunk cost fallacy, which, in the context of deep RL, is the tendency towards continuing an episode until termination. To address this, we propose learn to stop (LEAST), a lightweight mechanism that enables strategic early episode termination based on Q-value and gradient statistics, which helps agents recognize when to terminate unproductive episodes early. We demonstrate that our method improves learning efficiency on a variety of RL algorithms, evaluated on both the MuJoCo and DeepMind Control Suite benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。