arXiv:2604.26653cs.IR2026-04被引 2

构建可验证的智能体推理轨迹数据集,助力可信大模型训练

AgentSim: A Platform for Verifiable Agent-Trace Simulation

论文配图:AgentSim: A Platform for Verifiable Agent-Trace Simulation
图 1 · 摘自论文原文
  • 通过策略驱动探索文档集,生成分步可验证的推理过程
  • 构建超10万条推理步骤的ATC数据集,答案完全基于文档
  • 结合人机协作提升难例标注效率,适合模型行为研究

训练可信智能体大模型需要体现实际推理过程的数据,而非仅输出结果。现有数据集存在局限:问答数据仅含结果,思维链数据未关联具体文档,网页智能体数据追踪界面操作而非检索与整合核心步骤。本文提出AgentSim,一个用于模拟RAG智能体的开源平台,可生成任意文档集合上的可验证、分步推理轨迹。该平台采用策略确保智能体广泛探索文档集,并结合多模型验证流水线与主动式人工介入机制,将人力集中于模型分歧的难点环节。基于此,我们构建并发布Agent-Trace Corpus(ATC),包含超过103,000条覆盖三个主流信息检索基准的有依据推理步骤,实质性答案的溯源率为100%。本文贡献包括:(1) 提出AgentSim平台及两种机制——语料感知初始化与主动验证,提升轨迹多样性与质量;(2) 构建包含103,000+条可验证推理步骤的ATC数据集,覆盖三大IR基准,100%答案可追溯;(3) 通过对比分析揭示当前先进模型在信息获取策略上的系统性差异。平台、工具包与数据集均公开可用。

原文摘要 · Abstract (English)

Training trustworthy agentic LLMs requires data that shows the grounded reasoning process, not just the final answer. Existing datasets fall short: question-answering data is outcome-only, chain-of-thought data is not tied to specific documents, and web-agent datasets track interface actions rather than the core retrieval and synthesis steps of a RAG workflow. We introduce AgentSim, an open-source platform for simulating RAG agents. It generates verifiable, stepwise traces of agent reasoning over any document collection. AgentSim uses a policy to ensure the agent widely explores the document set. It combines a multi-model validation pipeline with an active human-in-the-loop process. This approach focuses human effort on difficult steps where models disagree. Using AgentSim, we construct and release the Agent-Trace Corpus (ATC), a large collection of grounded reasoning trajectories spanning three established IR benchmarks. We make three contributions: (1) the AgentSim platform with two mechanisms, Corpus-Aware Seeding and Active Validation, that improve trace diversity and quality; (2) the Agent-Trace Corpus (ATC), over 103,000 verifiable reasoning steps spanning three IR benchmarks, with 100% grounding rate on substantive answers; and (3) a comparative behavioral analysis revealing systematic differences in how state-of-the-art models approach information seeking. Platform, toolkit, and corpus are publicly available.

大模型推理数据RAG可验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。