用双流框架让2D模型读懂3D脑部CT,自动诊断急症病灶。
Brain-Adapter: A Dual-Stream Vision-Language MIL Framework for Comprehensive 3D CT Diagnosis of Acute Intracranial Pathologies

- 双流架构融合文本与视觉,用报告句子引导图像理解。
- 在10类急症病灶上达到92.3%平均AUC,优于现有3D模型。
- 适合临床部署,无需密集标注,可处理模糊病例。
3D脑部CT的自动化诊断对急症救治至关重要,但受限于人工标注成本高及传统模型语义理解弱。尽管2D基础视觉-语言模型(VLMs)表现优异,其表征能力难以有效迁移到3D体积数据。本文提出Brain-Adapter,一种新颖的双流多实例学习(MIL)框架,利用预训练2D生物医学VLM和原始诊断报告,实现鲁棒的扫描级多标签分类。我们引入文本条件注意力(TCA)机制,将原始诊断语句作为语义查询,动态对齐视觉线索与特定疾病概念;同时,平行的视觉MIL流捕捉全局扫描特征,由大语言模型(LLM)提取的结构化标签监督。为保证表示一致性,设计一致性约束强化双流协同。推理时,不确定性感知精炼(UAR)模块动态校准并融合双流预测,解决模糊病例。大量实验表明,该方法显著优于现有3D模型与标准MIL方法。通过消除对密集标注的依赖,Brain-Adapter为3D急性颅内病变分析提供高效可扩展、临床可用的解决方案。
原文摘要 · Abstract (English)
Automated diagnosis of 3D brain CT scans is essential for critical care, yet it remains challenging due to the heavy reliance on manual annotations and the limited semantic understanding of conventional models. While 2D foundation vision-language models (VLMs) have shown remarkable generalization, effectively transferring their representational power to 3D volumes remains an open problem. In this paper, we propose Brain-Adapter, a novel dual-stream multiple instance learning (MIL) framework that leverages pre-trained 2D biomedical VLMs and raw diagnostic reports for robust scan-level multi-label classification. Specifically, we introduce a Text-Conditioned Attention (TCA) mechanism, utilizing raw diagnostic sentences as semantic queries to dynamically align visual cues with specific disease concepts. Concurrently, a parallel visual MIL stream captures global scan characteristics, supervised by structured labels extracted via a Large Language Model (LLM). To ensure representation coherence, a consistency constraint enforces synergy between the two streams. During inference, an Uncertainty-Aware Refinement (UAR) module dynamically calibrates and fuses these dual-stream predictions to resolve ambiguous cases. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches. By eliminating the reliance on dense annotations, Brain-Adapter provides a highly scalable and clinically viable solution for 3D acute intracranial pathology analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。