通过采样后遗忘机制，提升大模型强化学习中的探索能力
EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
- 两阶段采样+轻量级遗忘，打破行为模式固化循环
- 在五个推理任务上相较GRPO平均提升24.3%~33.0%
- 适合需要持续探索的复杂决策场景，如大模型推理优化
在具有可验证奖励的大语言模型强化学习中，探索与利用的平衡仍是核心挑战。现有方法过度强调利用，导致熵坍塌、探索能力下降，性能提升受限。尽管提高策略随机性可促进探索，却常无法摆脱主导行为模式，形成重复采样并强化主导模式的自增强循环。本文提出探索增强型策略优化（EEPO），采用两阶段采样与自适应遗忘机制：第一阶段生成部分轨迹后，进行轻量级遗忘以暂时抑制已采样响应，迫使第二阶段探索输出空间的新区域。该‘采样-遗忘’机制有效打破自增强循环，增强采样过程中的探索广度。在五个推理基准测试中，EEPO显著优于GRPO：在Qwen2.5-3B上平均相对提升24.3%，在Llama3.2-3B-Instruct上提升33.0%，在Qwen3-8B-Base上提升10.4%。
原文摘要 · Abstract (English)
Balancing exploration and exploitation remains a central challenge in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). Current RLVR methods often overemphasize exploitation, leading to entropy collapse, diminished exploratory capacity, and ultimately limited performance gains. Although techniques that increase policy stochasticity can promote exploration, they frequently fail to escape dominant behavioral modes. This creates a self-reinforcing loop -- repeatedly sampling and rewarding dominant modes -- that further erodes exploration. We introduce Exploration-Enhanced Policy Optimization (EEPO), a framework that promotes exploration via two-stage rollouts with adaptive unlearning. In the first stage, the model generates half of the trajectories; it then undergoes a lightweight unlearning step to temporarily suppress these sampled responses, forcing the second stage to explore different regions of the output space. This sample-then-forget mechanism disrupts the self-reinforcing loop and promotes wider exploration during rollouts. Across five reasoning benchmarks, EEPO outperforms GRPO, achieving average relative gains of 24.3% on Qwen2.5-3B, 33.0% on Llama3.2-3B-Instruct, and 10.4% on Qwen3-8B-Base.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。