arXiv:2604.22989cs.CVcs.AI2026-04

CheXmix统一融合影像与文本生成,提升胸部X光诊断精度。

CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging

论文配图:CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging
图 1 · 摘自论文原文
  • 采用早期融合架构,直接联合处理图像和文本序列,避免特征扭曲。
  • 在高遮蔽率下分类任务AUROC提升8.6%,报告生成质量超基线45%。
  • 支持从粗到细的多任务学习,适合临床影像分析与生成场景。

近期医学多模态基础模型通常通过将预训练视觉编码器与语言模型连接,并使用类似LLaVA的方法微调,采用两阶段解耦策略,引入可能扭曲视觉特征的投影层。这在医学影像中尤为不利,因细微线索对准确诊断至关重要。相比之下,早期融合生成方法如Chameleon通过将图像与文本令牌统一处理,消除投影瓶颈,实现联合表示学习并利用语言模型的归纳先验。我们提出CheXmix,一种在大量胸片与放射科报告配对数据上训练的统一早期融合生成模型。在Chameleon自回归框架基础上,引入两阶段多模态生成预训练策略,结合掩码自编码器与多模态大模型的表征优势。模型具备高度灵活性,可支持粗粒度与细粒度的判别与生成任务。在所有遮蔽率下性能优于现有生成模型6.0%,在高遮蔽率下比CheXagent在CheXpert分类任务上提升8.6%(AUROC)。图像修复效果优于纯文本生成模型51.0%,在放射科报告生成的GREEN指标上超越CheXagent 45%。结果表明,CheXmix在广泛胸片任务中捕捉了细粒度信息。代码已开源:https://github.com/StanfordMIMI/CheXmix。

原文摘要 · Abstract (English)

Recent medical multimodal foundation models are built as multimodal LLMs (MLLMs) by connecting a CLIP-pretrained vision encoder to an LLM using LLaVA-style finetuning. This two-stage, decoupled approach introduces a projection layer that can distort visual features. This is especially concerning in medical imaging where subtle cues are essential for accurate diagnoses. In contrast, early-fusion generative approaches such as Chameleon eliminate the projection bottleneck by processing image and text tokens within a single unified sequence, enabling joint representation learning that leverages the inductive priors of language models. We present CheXmix, a unified early-fusion generative model trained on a large corpus of chest X-rays paired with radiology reports. We expand on Chameleon's autoregressive framework by introducing a two-stage multimodal generative pretraining strategy that combines the representational strengths of masked autoencoders with MLLMs. The resulting models are highly flexible, supporting both discriminative and generative tasks at both coarse and fine-grained scales. Our approach outperforms well-established generative models across all masking ratios by 6.0% and surpasses CheXagent by 8.6% on AUROC at high image masking ratios on the CheXpert classification task. We further inpaint images over 51.0% better than text-only generative models and outperform CheXagent by 45% on the GREEN metric for radiology report generation. These results demonstrate that CheXmix captures fine-grained information across a broad spectrum of chest X-ray tasks. Our code is at: https://github.com/StanfordMIMI/CheXmix.

医学影像多模态生成早期融合胸部X光

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。