用多模态大模型检测图像编辑差异，更贴近人类判断。
What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models
- 基于自回归多模态大模型构建差异检测与相关性评估模块。
- 在多个编辑模型上测试，与人类判断高度相关。
- 公开代码、模型和数据，支持可复现研究。
基于指令的图像编辑模型在生成任务中提供了更高的个性化潜力，但其结果评估仍具挑战性，现有指标在与人类判断的一致性和可解释性方面表现不足。为此，我们提出DICE（DIfference Coherence Estimator），一种用于检测原始图与编辑图之间局部差异，并评估其与修改请求相关性的模型。DICE由差异检测器和一致性评估器两部分构成，均基于自回归多模态大语言模型（MLLM），通过自监督、从修复网络蒸馏及全监督相结合的方式进行训练。通过大量实验，我们评估了框架内各阶段性能，比较了不同MLLM的表现。结果表明，DICE能有效识别出与指令一致的编辑内容，在多个编辑模型生成结果的评估中与人类判断具有强相关性。我们已公开发布源代码、模型及数据集。
原文摘要 · Abstract (English)
Instruction-based image editing models offer increased personalization opportunities in generative tasks. However, properly evaluating their results is challenging, and most of the existing metrics lag in terms of alignment with human judgment and explainability. To tackle these issues, we introduce DICE (DIfference Coherence Estimator), a model designed to detect localized differences between the original and the edited image and to assess their relevance to the given modification request. DICE consists of two key components: a difference detector and a coherence estimator, both built on an autoregressive Multimodal Large Language Model (MLLM) and trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision. Through extensive experiments, we evaluate each stage of our pipeline, comparing different MLLMs within the proposed framework. We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment. We publicly release our source code, models, and data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。