提出自动化可解释评估框架,精准衡量复杂指令下的创意图像编辑效果
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
- 基于问答机制构建全自动评估流程,克服大模型评分不透明问题
- 在超800个样本、1.3万条查询的基准上验证,闭源模型整体优于开源
- 评估结果与人工判断高度一致,适合研究创意图像生成与评测的学者
基于指令的多模态图像编辑近年来发展迅速,但现有评估方法缺乏系统化且符合人类认知的框架,难以有效评估模型在复杂和创意编辑任务中的表现。为此,我们提出CREval,一种完全自动化的基于问答(QA)的评估流程,克服了黑箱多模态大模型评分的不完整性和不可解释性。同时,我们构建了CREval-Bench,一个专为复杂指令下的创意图像编辑设计的综合性基准,涵盖三个类别和九个创意维度,包含超过800个编辑样本和13,000个评估问题。利用该流程与基准,我们系统评估了一系列先进开源与闭源模型。结果表明,尽管闭源模型在复杂创意任务中普遍表现更优,但所有模型仍难以有效完成此类编辑。此外,用户研究表明,CREval的自动化指标与人类判断具有强一致性。因此,CREval为复杂创意图像编辑任务提供了可靠评估基础,并指出了未来研究的关键挑战与机遇。
原文摘要 · Abstract (English)
Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。