让多模态大模型学会推断画面外的内容，提升视觉理解能力。
OpenView: Empowering MLLMs with Out-of-view VQA
- 构建四阶段流水线，用全景图像生成带空间上下文的多选视觉问答数据。
- 在合成数据上微调后，模型平均准确率从48.6%提升至64.1%。
- 提供可解释评估基准，适合研究视觉推理与模型泛化能力的学者。
当前多模态大模型在图像内容推理方面表现良好，但主要局限于可见画面内的信息。本文首次系统研究画面外（OOV）理解，即对视角范围外物体、活动和场景的推理能力。技术贡献有三：第一，设计OpenView四阶段流程，利用全景图像生成丰富上下文与空间定位的多选视觉问答数据；第二，构建高质量合成数据集OpenView-Dataset，基于真实世界全景图，支持模型监督微调；第三，建立OpenView-Bench评测基准，联合评估答案选择与推理过程的准确性，实现可解释性诊断。实验表明，尽管仍与人类表现存在差距，但经由OpenView增强后，多个主流模型性能显著提升，平均准确率从48.6%增至64.1%。代码、数据与评测平台将公开于https://github.com/q1xiangchen/OpenView。
原文摘要 · Abstract (English)
Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline to massively generate multi-choice VQA by leveraging panoramic imagery to enable context-rich and spatial-grounded VQA synthesis with free-view framing. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, upon empowered by OpenView, multiple MLLMs can consistently boost their performance, uplifted from 48.6% to 64.1% on average. Code, benchmark, and data will be available at https://github.com/q1xiangchen/OpenView.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。