让视觉问答模型学会识别无法回答的问题，避免胡乱猜测。
CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering
- 用CLIP的图文对齐能力判断问题是否可答，不改动原模型。
- 仅需训练少量新层，就能显著提升对不可答问题的识别率。
- 适合需要可靠答案的场景，如医疗或自动驾驶视觉系统。
视觉语言模型在视觉问答任务中表现卓越，但常会错误回答图像中不存在信息的不可答问题。为解决此问题，本文提出CLIP-UP：一种基于CLIP的轻量级不可答问题检测方法，使模型能主动拒绝回答不可答问题。该方法利用CLIP的图文相似度计算来提取问题与图像的对齐信息，仅需训练少量新增层，保持原有模型权重不变。在多个多选和开放式视觉问答基准上测试，CLIP-UP均显著优于现有方法，同时在其他任务上维持原有性能，展现出良好泛化能力。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) demonstrate remarkable capabilities in visual understanding and reasoning, such as in Visual Question Answering (VQA), where the model is asked a question related to a visual input. Still, these models can make distinctly unnatural errors, for example, providing (wrong) answers to unanswerable VQA questions, such as questions asking about objects that do not appear in the image. To address this issue, we propose CLIP-UP: CLIP-based Unanswerable Problem detection, a novel lightweight method for equipping VLMs with the ability to withhold answers to unanswerable questions. CLIP-UP leverages CLIP-based similarity measures to extract question-image alignment information to detect unanswerability, requiring efficient training of only a few additional layers, while keeping the original VLMs' weights unchanged. Tested across several models, CLIP-UP achieves significant improvements on benchmarks assessing unanswerability in both multiple-choice and open-ended VQA, surpassing other methods, while preserving original performance on other tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。