用代码实验自动化发现新智能体方法，产出19项成果
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation
- 将论文与代码块组合视为基因，进行跨文献的智能搜索
- 自动完成数百次实验，产出19项发现，6项经多重验证具增量创新性
- 适合对自动化科研、智能体设计感兴趣的开发者和研究者
尽管自主科学发现（ASD）在软件产物（如改进的机器学习算法）领域受到关注，现有系统仍面临两大局限：（1）主要在现有代码库或类似受限设计空间中探索变体；（2）生成大量研究产物（如自动生成的论文和代码），通常仅通过会议式评审评估，缺乏对代码的深入检验。本文提出CodeScientist，一种新型的自主科学发现系统，将构思与实验构建建模为在研究论文与定义领域通用操作（如提示语言模型）的代码块之间进行联合遗传搜索。该系统在智能体与虚拟环境领域内对数以百计的自动生成想法展开自动化实验，最终返回19项发现。其中6项经多维度评估（包括外部会议评审、代码审查与复现尝试）被认定至少具备基本合理性与增量新颖性。这些发现涵盖新任务、新智能体、新指标与新数据，表明从基准优化向更广泛发现的质变。
原文摘要 · Abstract (English)
Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。