用自动化流程筛选需专业软件计算的科学题,确保题目有效且有挑战性。
ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction
- 生成题目后通过三道关卡:可执行验证、无工具筛查、限时工具求解。
- 500次生成中保留135个有效题目,其中130个被工具成功解答。
- 适合需要真实计算验证的科学评测构建,节省专家人工审核时间。
科学基准测试通常由领域专家编写任务并相互核对,或改编教科书、论文与在线资源中的内容。此类方法虽能生成高质量评估,但每题需大量人工投入。语言模型可快速提出候选题目,但如何筛选仍是个难题。本文聚焦于答案需依赖专用软件计算而非仅靠推理的问题。候选题若脚本运行失败、结果不符或模型无需软件即可作答,则视为无效或过于简单。我们提出 ToolGate,将每个生成项视为提案,仅当三项关卡全部通过才予以保留:第一,执行脚本在科学软件下重现预期答案;第二,随机无工具筛查剔除模型仅凭提示就能解答的题目;第三,工具代理必须在固定时间内完成求解。我们在 FEniCSx 中实现 ToolGate,进行 500 次生成尝试。本地验证关卡保留 478 个候选。最终报告前再次筛查:两次随机无工具筛选排除 222 题,直接调用 GPT-5.5 API(默认中等推理)再排除 121 题。剩余 135 题中,使用 FEniCSx 的 GPT-5.5 Codex CLI 代理成功解决 130 题,经精确去重后剩 128 个唯一有效题。ToolGate 将重复的答案校验与难度筛选转化为可审计流程,同时将领域设计与最终评审交由专家完成。
原文摘要 · Abstract (English)
Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textbooks, published papers, and online resources. These routes can produce strong evaluations, but they require substantial per-item labor. Language models can reduce this repeated work by proposing candidates quickly. The remaining problem is acceptance. We target scientific questions whose answers require computations with specialist software rather than unaided reasoning alone. A candidate is invalid if its script fails or returns a different answer, or trivial if a model answers it without the software. We present ToolGate, which treats every generated item as a proposal and keeps it only if three gates pass. First, an executable solution script must reproduce the proposed answer when run with the scientific software. Second, randomized no-tool screening rejects candidates that models can already solve from the prompt alone. Third, a tool-using agent must solve each survivor within a fixed time limit. We instantiate ToolGate in FEniCSx with 500 generation attempts. The local-verification gate retains 478 candidates. For final reporting, we rescreen this pool after generation: two randomized no-tool screens exclude 222 from the reported pool, and direct GPT-5.5 API calls at medium reasoning (the API default) exclude another 121. Of the remaining 135, a GPT-5.5 Codex CLI agent with access to FEniCSx solves 130; exact deduplication leaves 128 unique protocol survivors. ToolGate turns repeated answer checking and difficulty screening into an auditable process while leaving domain design and final review to experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。