测试大模型在生物安全相关任务中的智能水平，发现其表现超过人类专家。
ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
- 构建多任务评估框架，涵盖实验设计、编程操控机器人和规避基因合成筛查。
- 所有测试模型在三项任务中均超越人类中位数水平，尤其在已有知识任务上表现优异。
- 适用于关注AI生物安全风险的研究者与政策制定者，揭示潜在滥用隐患。
大型语言模型（LLMs）正快速获得与生物研究相关的功能，从文献综述到实验数据分析，甚至能执行以往需资深生物学家完成的计算机模拟生物学任务。这些新兴的AI能力为科学发现和医学进步带来机遇，但也改变了生物安全风险格局。为此，我们提出「代理型生物能力基准」（ABC-Bench），一套用于评估代理类模型在生物安全相关任务上的能力。该基准涵盖良性与双用途生物学任务：编写代码控制液体处理机器人、设计体外组装的DNA片段，以及规避DNA合成审查。这些任务需要结合生物学与软件工程知识。所有测试的LLM代理在三项任务中均优于人类专家中位数。代理在依赖已发表知识和标准化流程的任务中表现良好，在需创新生物信息学推理的任务中较弱。在三次湿实验验证中，OpenAI的o4-mini-high生成的脚本在OpenTrons液体处理机器人上成功组装出预期序列的DNA。
原文摘要 · Abstract (English)
Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research, from literature synthesis to interpretation of experimental data. Increasingly, LLM agents can also perform in silico biology tasks that previously required experienced human biologists. These emerging AI capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks. To address this, we introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench), a suite of tasks to measure agentic biosecurity-relevant capabilities. ABC-Bench evaluates LLM agents on both benign and dual-use biology tasks: writing code to operate liquid handling robots, designing DNA fragments for in vitro assembly, and evading DNA synthesis screening. These tasks require a combination of biology and software expertise. All tested LLM agents outperformed the median expert human baseliner on all three tasks. Agents performed highly on tasks drawing on published knowledge and well-documented protocols, and more weakly on a task requiring novel bioinformatics reasoning. In three wet-lab validation experiments, we found that OpenAI's o4-mini-high produced scripts that, when run on an OpenTrons liquid handling robot, successfully assembled DNA with expected sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。