提出细粒度负向查询数据集，显著降低多模态大模型幻觉问题。
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
- 构建细粒度负向查询数据集FINER，针对多对象、多属性等复杂场景设计评测
- 在四项设置下发现模型在真实元素共存时仍会幻觉，且幻觉率显著上升
- 通过DPO微调提升4个前沿模型表现，幻觉减少最高达24.2%，通用能力也增强
多模态大语言模型（MLLMs）在细粒度查询下容易产生幻觉，现有基准主要关注粗粒度图像问题，忽视了这一挑战。本文提出细粒度负向查询（FINER）及其两个评测集：FINER-CompreCap与FINER-DOCCI。基于FINER，我们在多对象、多属性、多关系及“什么”类问题四种场景中分析幻觉现象。结果表明，当细粒度错误与真实存在的图像元素同时出现时，模型更易产生幻觉。为此，我们提出FINER-Tuning方法，利用直接偏好优化（DPO）在FINER启发的数据上进行微调。在四个前沿模型上应用FINER-Tuning后，在本研究的评测集上幻觉减少最多达24.2%（InternVL3.5-14B），同时在八个现有幻觉评测套件中性能提升，并在六个通用多模态基准上增强泛化能力。代码、数据集与模型已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2\% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at \href{https://explainableml.github.io/finer-project/}{https://explainableml.github.io/finer-project/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。