构建真实科研场景的多模态智能体评测平台,发现当前模型仅15%成功率。
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
- 设计动态可视化的跨领域科研工作流环境,支持多接口自主操作。
- 推出169个人工验证的真实科研任务,覆盖生化、天文等多学科。
- 揭示现有智能体在复杂流程中可靠性不足,为改进提供方向。
大语言模型(LLMs)已超越自然语言处理,推动跨学科研究发展。近期,基于LLM的智能体被用于协助科学发现的多个方面与领域。其中,能够像人类一样与操作系统交互的计算机使用型智能体,正推动科研任务自动化和研究流程优化。为释放其潜力,我们提出ScienceBoard,包含两项互补贡献:(i) 一个现实、多领域的动态视觉丰富科研工作流环境,集成专业软件,支持智能体通过多种界面自主交互以加速复杂研究任务;(ii) 由人类精心策划的169个高质量、严格验证的真实世界任务构成的挑战性基准,涵盖生物化学、天文学和地理信息学等领域的科学发现流程。对GPT-4o、Claude 3.7、UI-TARS等先进模型的广泛评估显示,尽管部分表现良好,但它们在复杂工作流中仍无法可靠辅助科学家,整体成功率仅为15%。深入分析揭示了当前智能体的局限性,并为更有效的设计原则提供了洞见,助力构建更具能力的科学发现智能体。代码、环境与基准详见https://qiushisun.github.io/ScienceBoard-Home/。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude 3.7, UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only a 15% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, environment, and benchmark are at https://qiushisun.github.io/ScienceBoard-Home/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。