用机制引导的LoRA提升医疗多模态模型对同义问题的回答一致性
Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
- 基于稀疏自编码器定位关键语义特征，指导LoRA微调以增强一致性
- 在MIMIC-CXR上问答翻转率从14.6%降至4.4%，置信度差减少79.5%
- 兼顾准确率稳定，适合需高一致性的临床辅助诊断系统
医疗视觉语言模型对同一临床问题的不同表述可能给出不同是/否回答。本文以MedGemma-4B为对象，使用PSF-Med数据集（包含158个二分类问题）评估其一致性。基线翻转率为14.6%，平均置信度差为1.63 logits。验证了Gemma Scope 2稀疏自编码器（SAEs）可有效迁移至MedGemma激活值，在医疗与通用文本上分别达到R² ≈ 0.997（n=100提示，p<0.001）。通过联合优化一致性与准确率的损失函数微调LoRA适配器，避免纯一致性训练导致的模式坍缩。在MIMIC-CXR上，翻转率降至4.4%（p=0.002），置信度差降至0.33（下降79.5%），准确率保持84.2%至82.3%（-1.9pp，不显著）。在PadChest Balanced（n=250）上，翻转率由13.6%降至7.8%，置信度差由1.08降至0.35（下降67.9%），准确率升至69.4%。层范围消融显示早期层比机制选定的中间层更有效降低置信度差。
原文摘要 · Abstract (English)
Medical Vision-Language Models can give different yes or no answers to rephrasings of the same clinical question. We study this in MedGemma-4B using PSF-Med Sadanandan and Behzadan (2025), which provides paraphrase pairs for systematic consistency evaluation on medical VQA. On MIMIC-CXR binary questions (n = 158), the baseline flip rate is 14.6% and mean margin difference is 1.63 logits. We validate that Gemma Scope 2 Sparse Autoencoders (SAEs) transfer to MedGemma activations, achieving R2 ~= 0.997 on both medical and general text (n = 100 prompts each, p < 0.001 for exceeding a 0.95 threshold). We then fine-tune Low-Rank Adaptation (LoRA) adapters with a combined loss that balances paraphrase consistency with answer accuracy. This combined approach prevents mode collapse that occurs with pure consistency training while reducing flip rate from 14.6% to 4.4% (p = 0.002, two-proportion z-test) and margin difference from 1.63 to 0.33 (79.5% reduction). Accuracy remains stable at 84.2% baseline versus 82.3% after training (-1.9pp, not significant). On PadChest Balanced (n = 250), flip rate drops from 13.6% to 7.8%, mean margin difference drops from 1.08 to 0.35 (67.9% reduction), and accuracy increases from 66.4% to 69.4%. A layer-range ablation shows that early layers reduce margin differences more than mechanistically selected middle layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。