用分层建模提升医学影像报告生成准确率
U-VLM: Hierarchical Vision Language Modeling for Report Generation
- 分阶段训练：从分割到分类再到报告生成，逐步提升
- 多层视觉注入：将编码器特征送入语言模型各层，保留多尺度信息
- 仅用0.1B小解码器就达顶尖性能，适合资源有限场景
自动化放射科报告生成有助于减轻放射科医生工作负担并提高诊断一致性，但针对3D医学影像生成准确报告仍具挑战。现有视觉-语言模型存在两大局限：未利用分割预训练编码器，且仅在语言模型输入层注入视觉特征，丢失多尺度信息。我们提出U-VLM，实现训练与架构上的分层视觉-语言建模：(1) 从分割到分类再到报告生成的渐进式训练；(2) 多层视觉注入，将U-Net编码器特征路由至对应语言模型层。各训练阶段可使用不同数据集，无需统一标注。U-VLM在CT-RATE上达到F1: 0.414（对比0.258）、BLEU-mean: 0.349（对比0.305），在AbdomenAtlas 3.0上分割检测F1达0.624（对比0.518），仅使用从零训练的0.1B解码器，证明精心设计的视觉编码器预训练优于7B以上预训练语言模型。消融实验表明，渐进式预训练显著提升F1，多层注入提升BLEU-mean。代码已开源。
原文摘要 · Abstract (English)
Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。