首个面向生物医学图像理解的大型语言扩散模型，性能超越现有方法。
LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
- 通过视觉指令微调，将语言扩散模型用于生物医学图像理解。
- 在三项VQA基准上创历史新高，最高达95.15%准确率。
- 能生成更长、更丰富的回答，适合需要详细解释的医疗场景。
自回归模型（ARMs）长期主导生物医学视觉语言模型（VLMs）领域。近期，如LLaDA等掩码扩散模型作为有前景的替代方案出现，但在生物医学领域的应用仍鲜有探索。为此，我们提出LLaDA-MedV，首个通过视觉指令微调针对生物医学图像理解优化的大规模语言扩散模型。LLaDA-MedV在开放式生物医学视觉对话任务中相比LLaVA-Med提升7.855%，相比LLaDA-V提升1.867%；在三项闭合形式VQA基准的子集上达到新纪录：VQA-RAD为84.93%，SLAKE为92.31%，PathVQA为95.15%。进一步分析表明，相较于LLaVA-Med，LLaDA-MedV可通过显式控制响应长度生成更长文本，从而输出更丰富信息。我们还深入分析了训练与推理阶段，强调初始化权重选择、微调策略以及采样步数与重复之间的相互作用至关重要。代码与模型权重已开源于https://github.com/LLM-VLM-GSL/LLaDA-MedV。
原文摘要 · Abstract (English)
Autoregressive models (ARMs) have long dominated the landscape of biomedical vision-language models (VLMs). Recently, masked diffusion models such as LLaDA have emerged as promising alternatives, yet their application in the biomedical domain remains largely underexplored. To bridge this gap, we introduce LLaDA-MedV, the first large language diffusion model tailored for biomedical image understanding through vision instruction tuning. LLaDA-MedV achieves relative performance gains of 7.855% over LLaVA-Med and 1.867% over LLaDA-V in the open-ended biomedical visual conversation task, and sets new state-of-the-art accuracy on the closed-form subset of three VQA benchmarks: 84.93% on VQA-RAD, 92.31% on SLAKE, and 95.15% on PathVQA. Furthermore, a detailed comparison with LLaVA-Med suggests that LLaDA-MedV is capable of generating reasonably longer responses by explicitly controlling response length, which can lead to more informative outputs. We also conduct an in-depth analysis of both the training and inference stages, highlighting the critical roles of initialization weight selection, fine-tuning strategies, and the interplay between sampling steps and response repetition. The code and model weight is released at https://github.com/LLM-VLM-GSL/LLaDA-MedV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。