小模型经精调可高效完成放射科问答，省时省力还准。
Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
- 用合成数据+分阶段微调，低成本训练30亿参数小模型
- 在开放与封闭式问题上表现媲美大模型，仅用有限数据
- 内置注意力热力图工具，医生可快速发现模型错误
近期视觉语言系统提升了放射科视觉问答（VQA）模型的准确性，但模型开发仍面临三重挑战：专家标注图像有限，难以大规模获取数据；放射影像模式复杂细微，建模难度高；缺乏有效评估机制，难识别模型误判情况。本研究对一个30亿参数的轻量级视觉语言模型进行微调，证明经过精心数据筛选的小模型在开放与封闭式问题上均能实现稳健表现。我们提出一种从合成问答对生成到专用放射学数据集（如ROCO v2.0、MedPix v2.0）多阶段微调的成本可控训练流程。结果表明，尽管参数量仅为顶尖模型（如LLaVA-Med）的一小部分，且训练数据规模有限，本模型仍取得令人满意的性能。我们还引入一种基于显著性图的轻量诊断工具，使领域专家可通过注意力分析检查模型表现并识别异常失效模式。
原文摘要 · Abstract (English)
Recent advancements in vision-language systems have improved the accuracy of Radiological Visual Question Answering (VQA) Models. However, some challenges remain across each stage of model development: limited expert-labeled images hinders data procurement at scale; the intricate and nuanced patterns of radiological images make modeling inherently difficult; and the lack of evaluation evaluation efforts makes it difficult to identify cases where the model might be ill-conditioned. In this study, we fine-tune a lightweight 3B parameter vision-language model for Radiological VQA, demonstrating that small models, when appropriately tuned with curated data, can achieve robust performance across both open- and closed-ended questions. We propose a cost-effective training pipeline from synthetic question-answer pair generation to multi-stage fine-tuning on specialised radiological domain-targeted datasets (e.g., ROCO v2.0, MedPix v2.0). Our results show that despite operating at a fraction of the scale of state-of-the-art models such as LLaVA-Med, our model achieves promising performance given its small parameter size and the limited scale of training data. We introduce a lightweight saliency-based diagnostic tool that enables domain experts to inspect VQA model performance and identify ill-conditioned failure modes through saliency analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。