无需标注数据,让搜索智能体自我进化提升推理能力
Dr. Zero: Self-Evolving Search Agents without Training Data
- 用自生成问题构建反馈循环,让提问与求解器互相促进
- 在多个问答基准上表现媲美甚至超过有监督方法
- 适合研究自主智能体、低数据依赖推理的学者
随着高质量数据获取日益困难,无需人工标注数据的自演化成为新范式。本文提出Dr. Zero框架,使搜索智能体仅依赖外部搜索引擎作为知识环境,实现无监督自演化。核心是设计一个自演化反馈循环:由提议者生成结构多样问题,训练源自同一基础模型的求解器;随着求解器进化,激励提议者生成更难但可解的任务,形成自动课程优化双代理。为提升效率,引入分组相对策略优化(HRPO),将结构相似问题聚类以建立群体基准,显著降低评估单个查询难度与可解性的采样开销。实验表明,Dr. Zero在多个问答基准上达到或超越全监督搜索智能体性能,证明强代理搜索与证据驱动推理可通过纯自演化涌现。
原文摘要 · Abstract (English)
As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。