构建首个放射科智能体基准，测试模型在真实影像环境中的诊断能力。
ABRA: Agent Benchmark for Radiology Applications
- 让智能体通过21种工具操作医学影像系统，模拟真实诊断流程。
- 仅89%任务执行成功，但结果正确率仅0-25%，暴露感知能力短板。
- 适合医疗AI研究者评估模型在复杂临床场景下的表现。
现有医学智能体基准仅提供预选图像样本，未将影像作为需自主导航的环境。我们提出ABRA，一个放射科智能体评测基准，智能体通过二十一种函数调用工具操作OHIF查看器与Orthanc DICOM服务器，涵盖切片导航、窗宽窗位调节、序列选择、像素坐标标注及结构化报告等功能。ABRA包含655个程序生成的任务，分为三个难度层级和八种类型（视图控制、元数据问答、视觉探测、标注、纵向对比、BI-RADS报告，以及标注和BI-RADS报告的模拟探测器变体），数据源自LIDC-IDRI、Duke Breast Cancer MRI和NLST New-Lesion LongCT。每个任务按规划、执行、结果三维度评分，采用任务类型特定的自动评分器。十款当前模型（五款闭源、五款开源）在真实标注任务中执行率达89%，但结果正确率仅为0-25%；而在模拟探测器提供的配对探测任务中，结果正确率可达69-100%，表明瓶颈在于感知而非工具调度。代码、任务生成器与评分器已开源：https://github.com/Luab/ABRA
原文摘要 · Abstract (English)
Existing medical-agent benchmarks deliver imaging as pre-selected samples, never as an environment the agent must navigate. We introduce ABRA, a radiology-agent benchmark in which the agent operates an OHIF viewer and an Orthanc DICOM server through twenty-one function-calling tools that span slice navigation, windowing, series selection, pixel-coordinate annotation, and structured reporting. ABRA contains 655 programmatically generated tasks across three difficulty tiers and eight types (viewer control, metadata QA, vision probe, annotation, longitudinal comparison, BI-RADS reporting, and oracle variants of annotation and BI-RADS reporting), drawn from LIDC-IDRI, Duke Breast Cancer MRI, and NLST New-Lesion LongCT. Each episode is scored along Planning, Execution, and Outcome (Bluethgen et al., 2025) by task-type-specific automatic scorers. Ten current models, five closed-weight and five open-weight, reach at least 89% Execution on real annotation but only 0-25% Outcome; on the paired oracle variant where a simulated detector supplies the finding, Outcome on the same task reaches 69-100% across the models evaluated, localising the bottleneck to perception rather than tool orchestration. Code, task generators, and scorers are released at https://github.com/Luab/ABRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。