arXiv:2606.05535cs.CVcs.AI2026-06

通过去噪自编码器提升医学图像问答的鲁棒性,减少噪声干扰。

Noise-Aware Visual Representation Learning for Medical Visual Question Answering

论文配图:Noise-Aware Visual Representation Learning for Medical Visual Question Answering
图 1 · 摘自论文原文
  • 在视觉编码前加入去噪自编码器,学习抗噪视觉表征。
  • 在SLAKE和PathVQA上验证,对噪声输入表现更稳定。
  • 采用LoRA实现高效微调,适合临床部署场景。

医学视觉问答(Med-VQA)通过让AI模型理解医学影像并回答临床问题,具有强大的临床决策支持潜力。现有方法通常使用轻量映射网络将现成的视觉编码器与大语言模型(LLM)连接,以降低计算成本。然而,这些方法常忽视视觉表征中噪声和微小无关变化的影响。为此,我们提出一种噪声感知的Med-VQA框架,在视觉嵌入映射到LLM输入空间前引入去噪自编码器。该自编码器预先训练以从污染输入重建干净视觉嵌入,促使模型学习对噪声不敏感的鲁棒视觉表征。随后通过多层感知机(MLP)将结果嵌入投影至语言模型嵌入空间,形成视觉前缀令牌,向LLM提供图像信息。为实现无需全量重训练的高效适配,采用低秩适应(LoRA)进行参数高效微调。在SLAKE和PathVQA基准上的实验表明,该方法在保持竞争性干净性能的同时,显著提升了对噪声输入的鲁棒性。结果表明,学习更鲁棒的视觉表征可有效增强Med-VQA的性能与稳定性。

原文摘要 · Abstract (English)

Medical visual question answering (Med-VQA) has strong potential for clinical decision support by enabling AI models to interpret medical images and answer clinically relevant queries. Recent approaches typically connect off-the-shelf vision encoders with large language models (LLMs) through lightweight mapping networks to reduce computational cost. However, these methods often overlook the importance of handling noise and small irrelevant changes in visual representations. To address these challenges, we propose a noise-aware Med-VQA framework that incorporates a denoising autoencoder before visual embeddings are mapped into the input space of an LLM. The denoising autoencoder is pretrained to reconstruct clean visual embeddings from corrupted inputs, encouraging the model to learn robust visual representations that are less sensitive to noise. The resulting embeddings are then projected into the language model embedding space using a multi-layer perceptron (MLP), forming visual prefix tokens that provide image information to the LLM. To enable efficient adaptation without full retraining, we employ parameter-efficient fine-tuning using low-rank adaptation (LoRA). The proposed method is evaluated on the SLAKE and PathVQA benchmarks. Experimental results show improved robustness to noisy input embeddings while maintaining competitive clean performance across multiple evaluation criteria. These findings suggest that learning more robust visual representations can enhance Med-VQA performance and robustness.

医学问答去噪鲁棒性LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。