用预测熵筛选医疗视觉语言模型的错误与重述不一致问题。
Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
- 用单次前向传播的预测熵识别易错和答案翻转样本。
- 在多个数据集上预测翻转的AUROC达0.82以上，且稳定可靠。
- 无需语义过滤，可直接用于临床部署，提升模型可信度。
医疗视觉语言模型在胸部X光片上回答二元存在性问题时，可能产生两种关联失败：自信出错，以及对临床上等价的问题重述后答案改变。这两种失败在二元输出头中均表现为相同的逻辑值差距，因此我们测试单一指标能否同时筛查二者。在MedGemma-4B-IT模型上，于MIMIC-CXR（分布内，n=196）和PadChest（分布外，n=861）数据集上，单次前向传播的预测熵对重述后答案翻转的预测性能分别达到AUROC 0.823和0.821，且在五个随机种子下稳定（0.828±0.021）。该信号在LLaVA-RAD-7B上也复现（AUROC 0.83），且对翻转定义（全部、操作者保留或否定排除）不变。一个低熵阈值能同时降低错误率（3.3%）和翻转率（5.7%），在80%覆盖率下实现。阈值选择可迁移至独立验证集。在跨站点数据分布偏移下，更昂贵的方法无实际增益：在5%风险目标下，单次前向传播可处理88.5%的案例，与蒙特卡洛丢弃（MC Dropout）相当，并在错误检测上比五种子模型平均集成高出0.111 AUROC。即使模型准确率达91%，仍有13%的样本在重述后自相矛盾，而单次前向传播即可识别。
原文摘要 · Abstract (English)
Medical Vision-Language Models (VLMs) answering binary presence questions on chest radiographs can fail in two linked ways: they are confidently wrong, and they change answers when a clinically equivalent question is rephrased. In a binary answer head both failures track the same logit margin, so we test how well one score screens for both. On MedGemma-4B-IT across MIMIC-CXR (in-distribution, n=196) and PadChest (out-of-distribution, n=861), single-pass predictive entropy predicts which yes/no answers flip under rephrasing with an area under the receiver operating characteristic curve (AUROC) of 0.823 on PadChest and 0.821 on MIMIC, and the signal is stable across five random seeds (0.828$\pm$0.021). The same signal replicates on LLaVA-RAD-7B (AUROC 0.83) and is invariant to the flip definition (all, operator-preserving, or negation-excluded rewrites), so the screen needs no semantic filter at inference. One low-entropy threshold drives error and flip rate down together (3.3% and 5.7% at 80% coverage), and threshold selection transfers to a held-out split. Under cross-site shift the more expensive methods we test add nothing practical: at the 5% risk target a single forward pass answers 88.5% of cases, matching Monte Carlo (MC) Dropout, and beats the uniformly averaged five-seed adapter ensemble on error detection by 0.111 AUROC. Even a model that is 91% accurate still contradicts itself on 13% of items under rephrasing, and one forward pass flags them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。