用变分贝叶斯让视觉问答模型更敢不答，减少幻觉。
Variational Visual Question Answering for Uncertainty-Aware Selective Prediction
- 引入变分贝叶斯方法，让模型在不确定时选择不回答
- 仅用一个后验样本就比AdamW训练的模型更可靠
- 适合对错误容忍度极低的高安全场景
尽管近年来视觉语言模型（VLMs）进展显著，但在视觉问答（VQA）和视觉推理任务中仍易出现过度自信和幻觉问题。贝叶斯方法可通过选择性预测提升可靠性——即仅在足够确定时才作答。然而，此类方法对大模型成本高且效果存疑。本文首次证明了变分贝叶斯在VQA中选择性预测的有效性与优势。我们基于深度学习中的变分方法，提出“变分VQA”（Variational VQA），显著改善模型校准性，在低误差容忍度（≤1%）条件下实现显著提升。实验表明，仅需一个后验样本生成的答案，就比使用AdamW训练的模型更可靠。此外，我们提出一种风险规避型选择器，通过考虑预测方差优于标准样本平均。整体而言，变分学习为大型VLMs的安全性和可信度提供了可行路径。
原文摘要 · Abstract (English)
Despite remarkable progress in recent years, Vision Language Models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models predict selectively, that is, models respond only when they are sufficiently confident. Unfortunately, such approaches can be costly and ineffective for large models, and there exists little evidence to show otherwise for multimodal applications. Here, we show for the first time the effectiveness and competitive edge of variational Bayes for selective prediction in VQA. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low ($\leq 1\%$). Often, just one posterior sample yields more reliable answers than those given by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。