用AI自动生成科学数据发现任务数据集,助力开源模型逼近GPT-4o性能。
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
- 利用大模型自动挖掘真实科研流程中的编码任务,构建高质量数据集。
- 新数据集包含5404个任务,覆盖4个学科,训练后模型在2个基准上性能翻倍。
- 适合研究科学智能、开源大模型优化及自动化科研工具的学者使用。
尽管长期致力于用AI加速科学发现,但构建AI共研者仍面临高质量数据稀缺的挑战。为解决此问题,我们提出AutoSDT,一个自动化的流水线,用于收集真实数据驱动科研工作流中的高质量编码任务。AutoSDT利用大模型的编码能力与参数化知识,搜索多样来源,筛选生态有效任务,并合成准确的任务指令与代码解决方案。通过该流程,我们构建了AutoSDT-5K数据集,包含5,404个数据驱动科学发现任务,覆盖四个科学领域和756个独特的Python包。据我们所知,这是首个自动构建且规模最大的开放数据集。专家对256个任务子集的评估显示:93%的任务具有生态有效性,92.2%的生成程序功能正确。在AutoSDT-5K上训练的Qwen2.5-Coder-Instruct系列模型(称为AutoSDT-Coder)在两个挑战性基准ScienceAgentBench和DiscoveryBench上表现显著提升。尤其在ScienceAgentBench上,AutoSDT-Coder-32B达到7.8%的成功率,与GPT-4o持平,是其基础模型性能的两倍;在DiscoveryBench上,假设匹配得分提升至8.1,相对改进17.4%,缩小了开源模型与GPT-4o之间的差距。
原文摘要 · Abstract (English)
Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。