arXiv:2603.23308cs.CVcs.AI2026-03被引 3

用视觉嫁接和分区压缩,让大模型更懂CT影像报告。

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

  • 分阶段课程学习,让语言模型逐步对接无文本训练的视觉特征。
  • 在CT-RATE数据集上宏平均F1达0.429,优于现有最优方法3.6%。
  • 适合医学影像生成、大模型视觉对齐方向的研究者参考。

从胸部CT体积自动生成自动化放射科报告极具挑战性,主要源于序列过长、类别严重失衡,以及大语言模型(LLMs)倾向于忽略视觉标记而依赖语言先验。本文提出Ker-VLJEPA-3B,一种四阶段课程学习框架,用于生成自由文本报告。其视觉主干(LeJEPA ViT-Large)在未标注的CT数据上通过自监督联合嵌入预测训练,无需文本监督,生成模态纯净的表示。不同于对比模型(如CLIP、BiomedCLIP),该设计避免了跨模态偏差。视觉-语言对齐被推迟至课程的桥接与生成阶段。方法创新包括:(1) 分区约束交叉注意力将切片嵌入压缩为32个空间定位的视觉标记;(2) 对各向异性LLM嵌入进行PCA白化;(3) 仅保留阳性发现的策略,防止后验崩溃;(4) 温启动桥接初始化,传递投影权重;(5) 选择性交叉注意力冻结结合弹性权重固化,防止灾难性遗忘。在CT-RATE基准测试中(2,984个验证体积,18类),该模型达到0.429的宏平均F1,优于当前最优(U-VLM,0.414)3.6%,阈值优化后达0.448(+8.2%)。消融实验表明,56.6%的生成质量源自患者特异性视觉内容。代码与权重已公开。

原文摘要 · Abstract (English)

Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.

CT报告生成视觉语言对齐课程学习自监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。