arXiv:2508.01287cs.LGcs.AI2025-08被引 2

只靠贪婪策略也能自发探索,关键在环境规律与记忆。

Exploitation Is All You Need... for Exploration

  • 用纯贪婪目标训练,利用环境规律和记忆实现自发探索。
  • 实验显示无结构或无记忆时探索行为消失,但信用分配缺失仍可保留探索。
  • 适合研究强化学习中探索机制的学者,尤其关注自适应策略设计者。

在训练元强化学习(meta-RL)代理解决新环境时,确保充分探索是一个核心挑战。传统方法通过随机化、不确定性奖励或内在奖励等显式激励来促进探索。本文提出假设:若满足三个条件,仅以贪婪(仅利用)为目标训练的智能体仍可能表现出涌现的探索行为——(1)重复性环境结构,即环境具有可重复的规律,使过往经验可指导未来决策;(2)智能体记忆,使其能保留并使用历史交互数据;(3)长时程信用分配,使学习能将延迟收益回传至当前决策。在随机多臂赌博机和时间延展网格世界中的实验表明,当结构与记忆共存时,纯贪婪策略会表现出信息寻求型探索行为。控制消融实验进一步证明,若缺少环境结构或记忆,探索行为即消失(条件1与2)。令人意外的是,移除长时程信用分配并不总导致探索失效——我们归因于伪汤普森采样效应。这些发现表明,在适当前提下,探索与利用不必作为对立目标,而可从统一的奖励最大化过程中自然涌现。

原文摘要 · Abstract (English)

Ensuring sufficient exploration is a central challenge when training meta-reinforcement learning (meta-RL) agents to solve novel environments. Conventional solutions to the exploration-exploitation dilemma inject explicit incentives such as randomization, uncertainty bonuses, or intrinsic rewards to encourage exploration. In this work, we hypothesize that an agent trained solely to maximize a greedy (exploitation-only) objective can nonetheless exhibit emergent exploratory behavior, provided three conditions are met: (1) Recurring Environmental Structure, where the environment features repeatable regularities that allow past experience to inform future choices; (2) Agent Memory, enabling the agent to retain and utilize historical interaction data; and (3) Long-Horizon Credit Assignment, where learning propagates returns over a time frame sufficient for the delayed benefits of exploration to inform current decisions. Through experiments in stochastic multi-armed bandits and temporally extended gridworlds, we observe that, when both structure and memory are present, a policy trained on a strictly greedy objective exhibits information-seeking exploratory behavior. We further demonstrate, through controlled ablations, that emergent exploration vanishes if either environmental structure or agent memory is absent (Conditions 1 & 2). Surprisingly, removing long-horizon credit assignment (Condition 3) does not always prevent emergent exploration-a result we attribute to the pseudo-Thompson Sampling effect. These findings suggest that, under the right prerequisites, exploration and exploitation need not be treated as orthogonal objectives but can emerge from a unified reward-maximization process.

强化学习探索机制元学习自适应策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。