用高保真模拟环境训练搜索智能体,成本低且效果优于真实网络交互。
SearchGym: Bootstrapping Real-World Search Agents via Cost-Effective and High-Fidelity Environment Simulation
- 构建可验证知识图谱与对齐文档库,确保任务事实准确、可解。
- 在9个基准上,Qwen2.5-7B模型平均比基线高10.6%。
- 适合想低成本训练强搜索能力智能体的研究者和开发者。
搜索智能体已成为解决开放性、知识密集型推理任务的关键范式。然而,通过强化学习(RL)训练这些智能体面临严峻困境:使用真实商业网页API交互成本过高,依赖静态数据快照又因数据错位引入噪声,导致奖励信号失真,使正确推理受罚或幻觉被奖励。为此,我们提出SearchGym,一个用于启动稳健搜索智能体的仿真环境。SearchGym采用严谨的生成流程构建可验证的知识图谱和对齐文档语料库,确保每个推理任务均有事实依据且严格可解。在此可控环境中,我们引入SearchGym-RL,一种渐进式课程学习方法,通过净化后的反馈逐步优化智能体策略,从基础交互演进至复杂长程规划。在Llama和Qwen系列模型上的大量实验表明,该方法具备优异的“仿真到现实”泛化能力。特别地,我们在SearchGym中训练的Qwen2.5-7B-Base模型,在九个多样化基准上平均超越基于网络增强的ASearcher基线10.6%。结果验证了高保真仿真是一种可扩展、高成本效益的搜索智能体开发路径。
原文摘要 · Abstract (English)
Search agents have emerged as a pivotal paradigm for solving open-ended, knowledge-intensive reasoning tasks. However, training these agents via Reinforcement Learning (RL) faces a critical dilemma: interacting with live commercial Web APIs is prohibitively expensive, while relying on static data snapshots often introduces noise due to data misalignment. This misalignment generates corrupted reward signals that destabilize training by penalizing correct reasoning or rewarding hallucination. To address this, we propose SearchGym, a simulation environment designed to bootstrap robust search agents. SearchGym employs a rigorous generative pipeline to construct a verifiable knowledge graph and an aligned document corpus, ensuring that every reasoning task is factually grounded and strictly solvable. Building on this controllable environment, we introduce SearchGym-RL, a curriculum learning methodology that progressively optimizes agent policies through purified feedback, evolving from basic interactions to complex, long-horizon planning. Extensive experiments across the Llama and Qwen families demonstrate strong Sim-to-Real generalization. Notably, our Qwen2.5-7B-Base model trained within SearchGym surpasses the web-enhanced ASearcher baseline across nine diverse benchmarks by an average relative margin of 10.6%. Our results validate that high-fidelity simulation serves as a scalable and highly cost-effective methodology for developing capable search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。