让大模型智能判断何时该探索，提升决策效率。
Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
- 基于变分推断设计细粒度奖励，评估探索动作的潜在价值。
- 在多个文本与图形界面任务中，显著优于传统探索策略。
- 适合需要动态决策的复杂智能体系统，如自主代理。
近期的智能体测试时扩展方法允许模型在做出最终决策前获取环境反馈。现有方法的主要局限在于采用统一的探索策略，无法根据不确定性自适应调整探索行为。本文提出一种探索感知的强化学习框架，使大语言模型智能体仅在高不确定性时主动探索。方法通过变分推断构建细粒度奖励函数，显式评估探索动作对后续决策的改进潜力，并引入探索感知分组机制，在优化过程中将探索性动作与任务完成动作分离。该设计聚焦信息缺口，实现选择性探索，并在任务上下文清晰后迅速转入执行阶段。实验表明，该方法在多个具有挑战性的文本与图形界面智能体基准测试中均取得稳定提升。代码与模型已公开于 https://github.com/HansenHua/EAPO-ICML26 及 https://huggingface.co/hansenhua/EAPO-ICML26。
原文摘要 · Abstract (English)
Recent advancements in agentic test-time scaling allow models to gather environmental feedback before committing to final actions. A key limitation of existing methods is that they typically employ undifferentiated exploration strategies, lacking the ability to adaptively distinguish when exploration is truly required. In this paper, we propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high. Our method introduces a fine-grained reward function via variational inference that explicitly evaluates exploratory actions by estimating their potential to improve future decision-making, together with an exploration-aware grouping mechanism that separates exploratory actions from task-completion actions during optimization. By targeting informational gaps, this design allows agents to explore selectively and transition to execution as soon as the task context is clear. Empirically, we demonstrate that our approach achieves consistent improvements across a range of challenging text-based and GUI-based agent benchmarks. Code is available at https://github.com/HansenHua/EAPO-ICML26 and models are available at https://huggingface.co/hansenhua/EAPO-ICML26.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。