用机器人实验室测试大模型在真实化学实验中的执行与适应能力。
Stress-testing large language model agents in a robotic chemistry laboratory
- 将45个模块化工作站开放为可机器读取的技能，构建物理世界测试环境。
- 仅3.3%的试验生成可执行流程，最优系统成功率28.1%，长程规划仍受限。
- 实验反馈仅触发局部调整，缺乏全局重规划，适合评估自主科研系统可靠性。
AI在知识、推理和计划生成方面已被广泛评估，但科学智能体还需具备可靠的物理操作能力及基于证据的适应性。本文利用机器人化学实验室作为物理世界测试平台，使科学智能体的执行能力可测量。45个模块化工作台以机器可读形式暴露为技能，共完成4,608次试验。仅有3.3%的试验在实验室约束下生成专家评估可执行的工作流，即使最佳系统也仅达28.1%。长周期规划仍具挑战：仅三个可执行流程超过30步操作，最长者达44步。经过五轮实验，反馈仅引发局部调整，未出现工作流级重规划或分析方法重构。本研究通过量化物理可执行性与证据驱动的重规划能力，提供了部署就绪性的实证评估，并建立闭环改进的诊断框架，推动真正物理扎根的自主科研发展。
原文摘要 · Abstract (English)
AI is evaluated through knowledge, reasoning and plan generation, yet scientific agency requires reliable physical action and adaptation to evidence. Here, we use a robotic chemistry laboratory as a physical-world testbed to make scientific agency measurable. Its 45 modular workstations exposed as machine-readable skills enabled 4,608 trials. Only 3.3% of trials produced expert-assessed executable workflows under laboratory constraints; even the best system achieved 28.1%. Long-horizon planning remained a challenge: only three executable workflows exceeded 30 operations, although the longest contained 44. Across five rounds, experimental feedback prompted local adjustments but no workflow-level replanning or analytical-method redesign. By making physical executability and evidence-driven replanning measurable, our study provides an evidence-based assessment of deployment readiness and a diagnostic framework to guide closed-loop improvements towards physically grounded autonomous research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。