arXiv:2606.01961cs.AI2026-06被引 3

首个面向医疗AI自主研究的全流程评估基准,可细粒度分析智能体行为。

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

论文配图:AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
图 1 · 摘自论文原文
  • 构建五阶段工作流框架,覆盖从规划到提交的完整研究流程。
  • 平均每轮33次交互,验证阶段表现最弱,提交失败占比超三成。
  • 适合研究自主医疗AI系统、智能体可靠性与错误机制的学者。

自主智能体正被期望支持端到端的医疗AI研究流程,超越单一预测任务或短文本临床问答。然而,现有医疗智能体基准多仅评估最终输出,难以揭示研究过程中的行为细节。为此,我们提出AutoMedBench,一个面向多样化医学影像与多模态推理任务的流程感知型基准,将智能体执行组织为统一的五阶段工作流(S1-S5):计划、准备、验证、推断与提交。该基准包含长周期任务,每轮平均33次智能体交互,涵盖五个研究方向:分割、图像增强、视觉问答(VQA)、报告生成与病灶检测。每项任务设轻量级(Lite)与标准级(Standard)两档难度,使用相同数据与指标但提供不同任务简述支持。每轮评估同时包含最终任务性能与各阶段评分,实现从任务简述到最终提交的逐阶段分析。在数千次记录运行中,阶段评分显示验证阶段平均表现最差,而准备阶段最强,表明当前智能体更擅长构建可执行流程,而非验证其可靠性。事后错误分析显示,验证与提交失败是主要错误类型,分别占触发代码的37.7%和38.1%,任务理解错误罕见(0.9%),且单次触发错误的运行整体得分比无错误运行低48%。

原文摘要 · Abstract (English)

Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.

医疗AI自主智能体评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。