arXiv:2608.03322cs.CV2026-08

构建20万条医学多模态图文定位数据集,提升AI对医疗影像的精准理解

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

论文配图:LocAnyMed: Vision-Language Grounding for Multimodal Medical Images
图 1 · 摘自论文原文
  • 构建统一格式的医学图文定位数据集,覆盖多种影像模态
  • 在新数据集上微调后,定位准确率从10.64提升至85.59
  • 新增可解释推理链,帮助医生理解AI判断依据

医学视觉定位将自由形式的临床问题与医学图像中的空间证据关联,是可解释医疗人工智能的重要组成部分。然而,通用定位模型主要在自然图像上训练,现有医学定位资源分散于不同成像模态、数据集和任务形式中。为此,我们构建了包含约20万张图像-查询-答案对的LocAnyMed-200K多模态医学视觉定位数据集,覆盖计算机断层扫描(CT)、光学医学成像、超声和X射线。我们将异构的检测与定位资源统一为支持单个或多个边界框、点坐标及无目标输出的自由形式指令格式。在LocAnyMed-200K上对LocateAnything-3B进行全参数微调,使其在保留评估集上的F1@IoU 0.50从10.64提升至85.59,证明大规模领域特定监督可有效赋予通用模型医学定位能力。除空间坐标外,临床可解释系统还需说明预测依据。因此,我们构建了包含2万条推理链的LocAnyMed-CoT-20K子集,通过结构化推理连接解剖背景、视觉观察与空间结论,并进一步提升跨源泛化性能。这些资源共同为研究异构医学成像模态下的定位精度与推理质量提供了统一基础。代码已公开于https://github.com/MiliLab/LocAnyMed。

原文摘要 · Abstract (English)

Medical visual grounding connects free-form clinical queries to spatial evidence in medical images and is an important component of interpretable medical artificial intelligence. However, general-purpose grounding models are predominantly trained on natural images, while existing medical localization resources remain fragmented across imaging modalities, datasets, and task formulations. To address this gap, we construct LocAnyMed-200K, a multimodal medical visual grounding dataset containing approximately 200K image-query-answer examples across computed tomography, optical medical imaging, ultrasound, and X-ray. We harmonize heterogeneous detection and localization resources into a unified free-form instruction format that supports one or multiple bounding boxes, point coordinates, and no-target outputs for negative queries. Full-parameter fine-tuning of LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on a held-out evaluation split, demonstrating that large-scale domain-specific supervision can equip a general grounding model with effective medical localization capabilities. Beyond spatial coordinates, a clinically interpretable grounding system should also communicate the evidence supporting its prediction. We therefore derive LocAnyMed-CoT-20K, a rationale-augmented subset that connects anatomical context, visual observations, and spatial conclusions through structured reasoning and further improves cross-source generalization through fine-tuning. Together, these resources provide a unified foundation for studying both localization accuracy and rationale quality across heterogeneous medical imaging modalities. The code is publicly available at https://github.com/MiliLab/LocAnyMed.

医学影像图文定位可解释AI多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。