提出细粒度幻觉分类框架，提升图像重置任务中的错误检测精度。
VIGIL: Tackling Hallucination Detection in Image Recontextualization
- 将幻觉分为五类，针对性设计多阶段检测流程。
- 在1269张标注样本上实现开源模型最佳宏平均F1值。
- 可生成错误文本解释，适合模型调试与评估场景。
我们提出VIGIL（视觉不一致与生成上下文清晰性），一个用于大型多模态模型图像重置任务中幻觉检测的基准数据集与框架。现有方法通常将幻觉视为单一错误类型，而我们将其细分为五类：物体视觉保真度、背景保真度、空间与指令保真度、物理与融合保真度以及物体遗漏。我们构建了一个多阶段检测流水线，通过一组开源模型协同处理这五类问题。在涵盖五个产品领域的1,269个手动标注样本的VIGIL数据集上进行评估，报告各类别宏平均F1。该分解式流水线在开源检测器中表现最优，并能为每项检测到的错误提供文本解释，这是此前方法所不具备的能力。我们已通过GitHub（https://github.com/mlubneuskaya/vigil）和Hugging Face数据仓库（https://huggingface.co/datasets/joannaww/VIGIL）公开发布VIGIL数据集、检测流水线及基准代码。
原文摘要 · Abstract (English)
We introduce VIGIL (Visual Inconsistency & Generative In-context Lucidity), a benchmark dataset and framework that provides a fine-grained categorization of hallucinations in the multimodal image recontextualization task for large multimodal models. Most existing methods treat hallucinations as a single undifferentiated error. We instead decompose them into five categories, namely Object Visual Fidelity, Background Fidelity, Spatial and Instructional Fidelity, Physical and Integration Fidelity, and Object Omission. We propose a multi-stage detection pipeline that processes recontextualized images through specialized steps targeting all five categories with a coordinated set of open-source models. We evaluate the pipeline on the VIGIL dataset of 1,269 manually annotated samples across five product domains and report macro-F1 per category. The decomposed pipeline reaches the best macro-F1 among open-source detectors. It returns a textual explanation for each detected error, which prior methods for this task do not provide. We openly release VIGIL, along with the detection pipeline and benchmark code, through our GitHub repository: https://github.com/mlubneuskaya/vigil and Data repository: https://huggingface.co/datasets/joannaww/VIGIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。