AI科学家缺乏执行验证能力,制约了突破性成果产出。
AI Scientists Fail Without Strong Implementation Capability
- 指出当前AI科学家在实验执行与验证环节存在根本短板
- 28篇论文评估显示系统无法完成严谨实验验证
- 呼吁学界重视实现能力,推动科研自动化落地
人工智能科学家正引领科学发现的新范式,大型语言模型(LLMs)主导从构想到实验的全流程。尽管近期研究已能生成获ICLR 2025与ACL 2025接收的研究报告,宣称接近人类水平的自主发现能力,但其在计算机科学领域尚未产生可媲美自动化工具的突破性成果。基于对复杂工程任务基准的大量定量分析及对五种先进AI科学家系统生成的28篇论文的系统评估,本文认为:AI科学家的根本瓶颈在于执行必要验证程序的能力不足。现有系统缺乏完成严谨实验并产出高质量论文所需的执行能力。为揭示这一‘实现差距’的根源,本文深入探讨了AI科学家的基本局限性。本立场论文旨在呼吁社区共同努力,弥合实现差距。
原文摘要 · Abstract (English)
The emergence of Artificial Intelligence (AI) Scientist represents a paradigm shift in scientific discovery, with large language models (LLMs) taking the lead as the primary executor in the entire scientific workflow from idea generation to experiment implementation. Recent AI Scientist studies demonstrate sufficient capabilities for independent scientific discovery, with the generated research reports gaining acceptance at the ICLR 2025 workshop and ACL 2025, arguing that a human-level AI Scientist, capable of uncovering phenomena previously unknown to humans, may be imminent. Despite this substantial progress, AI Scientist has yet to produce a groundbreaking achievement in the domain of computer science on par with automated scientific tools. Based on extensive quantitative evidence from existing benchmarks in complex engineering tasks and a systematic evaluation assess 28 research papers generated by five advanced AI Scientist systems, we argue that \textbf{the fundamental bottleneck for AI Scientists lies in their capability to execute the requisite verification procedures.} Current AI Scientist systems lack the execution capabilities needed to execute rigorous experiments and produce high-quality scientific papers. To better illustrate the root cause of this \textbf{implementation gap}, we provide an in-depth discussion on the fundamental limitations of AI Scientist. This position paper aims to call for the participants in the community to bridge the implementation gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。