arXiv:2601.02737cs.CV2026-01被引 3

发现并解决医学影像模型对功能信息感知的缺失问题。

Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench

  • 提出原子级视觉对齐方法,先学功能特征再推理诊断。
  • 在5万+问答对上测试,诊断准确率提升14.83%。
  • 适合医疗AI研究者和临床辅助系统开发者。

尽管多模态大语言模型在解剖影像的异常检测与报告生成中表现优异,但其在功能影像方面的能力仍鲜有探索。本文识别并量化了一个根本性功能感知差距:当前视觉编码器无法在无解剖先验的情况下解析功能示踪剂分布。以正电子发射断层扫描(PET)为典型模态,我们构建了首个大规模功能影像基准PET-Bench,包含来自9,732例多中心、多示踪剂PET研究的52,308个分层问答对。对19种先进多模态大模型的评估揭示了一种称为链式思维(CoT)幻觉陷阱的关键安全风险:标准CoT提示虽增强语言流畅性,却使诊断脱离视觉证据,生成看似合理实则错误的结论。为此,我们提出原子级视觉对齐(AVA),一种通过细调强制低层级功能感知优先于高层级推理的策略。实验表明,AVA有效弥合感知鸿沟,将CoT从幻觉源头转变为可靠推理工具,诊断准确率最高提升14.83%。代码与数据见https://github.com/yezanting/PET-Bench。

原文摘要 · Abstract (English)

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in tasks such as abnormality detection and report generation for anatomical modalities, their capability in functional imaging remains largely unexplored. In this work, we identify and quantify a fundamental functional perception gap: the inability of current vision encoders to decode functional tracer biodistribution independent of morphological priors. Identifying Positron Emission Tomography (PET) as the quintessential modality to investigate this disconnect, we introduce PET-Bench, the first large-scale functional imaging benchmark comprising 52,308 hierarchical QA pairs from 9,732 multi-site, multi-tracer PET studies. Extensive evaluation of 19 state-of-the-art MLLMs reveals a critical safety hazard termed the Chain-of-Thought (CoT) hallucination trap. We observe that standard CoT prompting, widely considered to enhance reasoning, paradoxically decouples linguistic generation from visual evidence in PET, producing clinically fluent but factually ungrounded diagnoses. To resolve this, we propose Atomic Visual Alignment (AVA), a simple fine-tuning strategy that enforces the mastery of low-level functional perception prior to high-level diagnostic reasoning. Our results demonstrate that AVA effectively bridges the perception gap, transforming CoT from a source of hallucination into a robust inference tool and improving diagnostic accuracy by up to 14.83%. Code and data are available at https://github.com/yezanting/PET-Bench.

医学影像多模态模型功能成像视觉对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。