首个评测AI智能体自主发现数据集的基准,揭示当前系统仅22%成功率。
DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discovery
- 构建208个真实需求的多维评估框架,测试智能体发现与合成数据能力。
- 顶尖系统在挑战集上仅达22%准确率,暴露出严重泛化缺陷。
- 揭示检索型与生成型智能体各有优劣,但均难应对罕见场景。
大语言模型的发展使人工智能的瓶颈从算力转向数据可用性——大量有价值的数据集隐藏于专业库、研究附录和领域平台中。随着推理能力与深度研究方法的进步,一个关键问题浮现:能否让AI智能体超越传统搜索,系统性地发现满足特定用户需求的数据集,实现真正的自主数据定制?我们提出DatasetResearch,首个全面评估智能体在知识密集型与推理密集型任务中发现并合成数据集能力的基准。其三维评估框架显示:即使最先进的深度研究系统,在挑战性子集DatasetResearch-pro上也仅取得22%的得分,暴露出当前能力与理想数据发现之间的巨大差距。分析揭示根本矛盾:搜索类智能体在知识任务中靠广度检索表现优异,而合成类智能体通过结构化生成主导推理任务,但两者在分布外的‘边缘案例’上均彻底失败。这些发现确立了首个数据发现智能体的严格基准,并指明迈向能从数字宇宙中找到任何数据的AI系统的路径。该基准与完整分析已公开于https://github.com/GAIR-NLP/DatasetResearch。
原文摘要 · Abstract (English)
The rapid advancement of large language models has fundamentally shifted the bottleneck in AI development from computational power to data availability-with countless valuable datasets remaining hidden across specialized repositories, research appendices, and domain platforms. As reasoning capabilities and deep research methodologies continue to evolve, a critical question emerges: can AI agents transcend conventional search to systematically discover any dataset that meets specific user requirements, enabling truly autonomous demand-driven data curation? We introduce DatasetResearch, the first comprehensive benchmark evaluating AI agents' ability to discover and synthesize datasets from 208 real-world demands across knowledge-intensive and reasoning-intensive tasks. Our tri-dimensional evaluation framework reveals a stark reality: even advanced deep research systems achieve only 22% score on our challenging DatasetResearch-pro subset, exposing the vast gap between current capabilities and perfect dataset discovery. Our analysis uncovers a fundamental dichotomy-search agents excel at knowledge tasks through retrieval breadth, while synthesis agents dominate reasoning challenges via structured generation-yet both catastrophically fail on "corner cases" outside existing distributions. These findings establish the first rigorous baseline for dataset discovery agents and illuminate the path toward AI systems capable of finding any dataset in the digital universe. Our benchmark and comprehensive analysis provide the foundation for the next generation of self-improving AI systems and are publicly available at https://github.com/GAIR-NLP/DatasetResearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。