让大模型在搜索任务中自我进化，不依赖人工数据或固定规则。
Towards Agentic Self-Learning LLMs in Search Environment
- 用生成式奖励模型替代固定规则，实现开放域学习
- 增加任务数据量（含合成数据）显著提升智能体能力
- 多角色协同进化框架支持持续自提升，适合零标注数据场景
我们研究了在不依赖人工标注数据或预设规则奖励的情况下，基于大模型的智能体能否实现可扩展的自学习。通过在搜索代理场景中的受控实验，我们发现可扩展训练的两个关键因素：奖励信号来源和智能体任务数据规模。结果表明，生成式奖励模型（GRM）的奖励优于固定规则信号，且与策略共同演化能进一步提升性能；增加任务数据量——即使为合成生成——也能显著增强智能体能力。基于此，我们提出全闭环、多角色强化学习框架「Agentic Self-Learning」（ASL），统一任务生成、策略执行与评估，在共享工具环境与大模型主干下形成更难任务、更准验证、更强求解的良性循环。实证显示，ASL实现持续迭代提升，超越强基线（如Search-R1）且不退化，在零标注数据条件下仍持续进步，体现优异样本效率与鲁棒性。我们进一步揭示GRM验证能力是主要瓶颈：若冻结则引发奖励欺骗并停滞进展；持续在动态数据上训练GRM可缓解此问题，少量后期真实验证数据注入可突破性能上限。本工作确立奖励来源与数据规模为开放域智能体学习的关键杠杆，验证多角色协同进化的有效性。代码与数据已开源。
原文摘要 · Abstract (English)
We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data-even when synthetically generated-substantially enhances agentic capabilities. Building on these insights, we propose \textbf{Agentic Self-Learning} (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。