arXiv:2606.06534eess.IVcs.AI2026-06中稿 · CVPR

通过轻量配准与注意力引导,提升医学影像纵向问答的准确性和可解释性。

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

论文配图:Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models
图 1 · 摘自论文原文
  • 用轻量仿射配准对齐不同时期图像,减少运动干扰。
  • 在Medical-Diff-VQA上达到BLEU、ROUGE-L等指标领先结果。
  • 结合自监督与监督目标,适配视觉基础模型在医学领域的应用。

纵向医学视觉问答(VQA)需要推理当前时间点图像与参考时间点图像之间的解剖差异。本文提出一种注意力引导的编码器-解码器框架,用于胸部X光片任务。不同于传统的直接对比,我们引入一个轻量级仿射配准模块,通过小规模正则化将当前图像配准到参考图像,以减少无关运动。配准后的图像对输入图像编码器,随后接入冻结的DINO-based掩码生成器和可训练的自适应掩码生成器,生成作用于原始图像对的掩码。掩码后的图像对再次输入图像编码器,并与文本特征拼接后输入多模态变压器解码器,生成最终答案。为促进学习稳定并明确变化信号,受DINO-v3启发,引入辅助目标:掩码重建损失、成对格拉姆风格一致性损失和KoLeo均匀性损失,增强表征几何结构。在Medical-Diff-VQA基准测试中,模型在BLEU、ROUGE-L、CIDEr和METEOR等指标上表现优异,且通过共享显著性掩码实现内在可解释性。结果表明,基于显著性条件生成与温和预对齐是一种适用于医学纵向推理的合理框架。训练策略也展示了利用视觉基础模型在生物医学中联合优化监督与无监督目标的潜力。

原文摘要 · Abstract (English)

Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point. We propose an attention-guided encoder-decoder for this task with chest X-rays. Instead of conventional direct contrast, we propose to include a lightweight affine registration module to reduce nuisance motion by co-registering the current image to the reference image with a small registration regularizer. The registered image pair is fed into the image encoder, followed by a frozen DINO-based mask generator and a trainable adaptive mask generator to produce masks applied to the original image pairs. The masked image pairs are again fed into the image encoder and concatenated with text features as the input to a multimodal transformer-based decoder to generate final answers. To facilitate learning stabilization and clarify the change signal, inspired by DINO-v3, we include additional auxiliary objectives, including a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss, which enhances the geometry of the representation. On the Medical-Diff-VQA benchmark, the model delivers strong BLEU, ROUGE-L, CIDEr, and METEOR scores while offering intrinsic interpretability through the shared saliency mask. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA. Our training strategy also illustrates the potential of a paradigm in utilizing image foundation models in biomedicine: optimizing both supervised and unsupervised learning objectives simultaneously.

医学VQA纵向分析视觉基础模型可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。