测试大模型在异常场景下的物体识别能力，发现上下文不符会大幅降低准确率。
ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models
- 用LLM和CLIP生成异常上下文中的物体对，构建测试集
- 18个模型在异常场景下性能显著下降，最高误差超40%
- 适合研究模型鲁棒性与可信AI的开发者使用
大型视觉语言模型（LVLMs）在图像描述、视觉问答和机器人任务中表现优异，但在非常规场景中常遗漏明显物体或产生虚构物体。本文从不确定性角度出发，聚焦上下文不符问题——即物体出现在不合逻辑的场景中或应出现却未出现，发现此类情况显著增加当前先进LVLMs的识别难度。为此，提出物体在异常上下文中的识别框架ORIC，通过两种互补策略构建异常对象-场景对：(1) 利用大语言模型引导采样，识别图像中存在但难以识别的物体；(2) 利用CLIP引导采样，挖掘合理但实际缺失的物体。该方法应用于MSCOCO数据集，生成ORIC-Bench与ORIC风格训练数据。评估18个LVLMs及2个开放词汇检测器显示，在异常上下文中性能显著下降并呈现偏差。对Qwen3-VL-8B-Instruct在600个ORIC样本上进行视觉强化微调后，其在ORIC-Bench、AMBER和HallusionBench上的表现均有提升。总体表明，上下文不符是不确定性的重要来源，并提供了促进更可靠LVLMs的工具。数据集与代码已公开于https://github.com/ZhaoyangLi-1/ORIC。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) excel at captioning, visual question answering, and robotics by combining vision and language, yet they often miss obvious objects or hallucinate nonexistent ones in atypical scenes. We examine these failures through the lens of uncertainty, focusing on contextual incongruity, where objects appear unexpectedly or fail to appear in expected contexts, and show that such cases increase recognition difficulty for state-of-the-art LVLMs. To study this regime, we introduce the Object Recognition in Incongruous Context (ORIC) framework, which constructs incongruous object-context pairs through two complementary strategies: (1) LLM-guided sampling to identify hard-to-recognize objects present in the image and (2) CLIP-guided sampling to mine plausible but absent ones. Applied to MSCOCO, ORIC creates ORIC-Bench and ORIC-style training data. Evaluating 18 LVLMs and 2 open-vocabulary detectors reveals significant degradation and bias under incongruous contexts. Visual Reinforcement Fine-Tuning of Qwen3-VL-8B-Instruct on 600 ORIC samples improves performance on ORIC-Bench, AMBER, and HallusionBench. Overall, we show that contextual incongruity is a key source of uncertainty and provide tools for more reliable LVLMs. The dataset and code are publicly available at https://github.com/ZhaoyangLi-1/ORIC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。