首个可验证的科学数据发现环境数据集,助力大模型真实科研任务训练
D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery
- 自动构建跨学科科学任务环境,支持指令+可执行代码+评测脚本
- 评测脚本与人工标注一致率达87.5%,显著提升模型在ScienceAgentBench上的表现
- 适合研究智能体科研优化循环,开源全部数据与流程供复现
尽管语言模型和智能体在数据驱动的科学发现中取得进展,但其能力受限于缺乏可验证的真实世界科学任务环境。为此,我们提出D3-Gym,首个自动生成的、可用于科学数据驱动发现的可验证环境数据集。D3-Gym涵盖来自239个真实科学仓库的565个任务,覆盖四个学科,每个任务包含自然语言指令、预装依赖的可执行环境、数据集预览、参考解法及自动生成的评估脚本。评估脚本与人工标注金标准达成87.5%的一致性,并在领域特定评估逻辑上表现出强一致性。在D3-Gym轨迹上训练Qwen3系列模型,在ScienceAgentBench上实现稳定提升,其中Qwen3-32B提升7.8个百分点,缩小了与主流专有模型的差距。案例研究进一步展示了D3-Gym在真实科研工作流中研究自研优化循环(如Autoresearch)的潜力。项目已开源,包括数据集、构建流程、采样轨迹和训练脚本,详见https://github.com/OSU-NLP-Group/D3-Gym。
原文摘要 · Abstract (English)
Despite recent progress in language models and agents for scientific data-driven discovery, advancing their capabilities is held back by the absence of verifiable environments representing real-world scientific tasks. To fill this gap, we introduce D3-Gym, the first automatically constructed dataset with verifiable environments for scientific Data-Driven Discovery. D3-Gym comprises 565 tasks from 239 real scientific repositories across four disciplines, each with a natural language instruction, an executable environment with pre-installed dependencies, dataset previews, a reference solution, and an automatically synthesized evaluation script. Our evaluation scripts achieve 87.5% agreement with human-annotated gold standards and strong alignment in domain-specific evaluation logic. Training on trajectories sampled from D3-Gym yields consistent gains across Qwen3 models on ScienceAgentBench, boosting Qwen3-32B by 7.8 absolute points and shrinking the gap with strong proprietary models. We further illustrate, through case studies, how D3-Gym environments can serve as a testbed for studying agentic optimization loops such as Autoresearch on real scientific workflows. We open-source D3-Gym, its creation workflow, sampled trajectories, and training scripts at https://github.com/OSU-NLP-Group/D3-Gym.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。