构建首个覆盖创意全过程的多模态创造力评估基准
CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product
- 从创意构想到成果呈现，多维度评估人类定义的创造力
- 基于2.2万条多模态数据和470万条指令训练出更贴近人类判断的模型
- 适合需要精准评估创意能力的研究者与开发者使用
人类定义的创造力高度抽象，给多模态大语言模型（MLLMs）理解并评估符合人类判断的创造力带来挑战。现有基准缺失进一步加剧这一困境。为此，我们提出CreBench，包含两个核心组件：1）覆盖创意构思、过程到成果的多维度评估基准；2）CreMIT（创造力多模态指令微调数据集），包含2.2K来源多样化的多模态数据、79.2K条人工反馈和470万条多类型指令。为提升MLLM对多样化创意问题的处理能力，我们利用GPT对人工反馈进行优化，以激活更强的创造力评估能力。CreBench为构建理解人类对齐创造力的MLLM奠定基础。基于此，我们微调开源通用MLLM，得到CreExpert——一个专注于多模态创造力评估的专家模型。大量实验表明，所提CreExpert模型在与人类创造力评价的一致性上显著优于当前最先进的MLLMs，包括最前沿的GPT-4V和Gemini-Pro-Vision。
原文摘要 · Abstract (English)
Human-defined creativity is highly abstract, posing a challenge for multimodal large language models (MLLMs) to comprehend and assess creativity that aligns with human judgments. The absence of an existing benchmark further exacerbates this dilemma. To this end, we propose CreBench, which consists of two key components: 1) an evaluation benchmark covering the multiple dimensions from creative idea to process to products; 2) CreMIT (Creativity Multimodal Instruction Tuning dataset), a multimodal creativity evaluation dataset, consisting of 2.2K diverse-sourced multimodal data, 79.2K human feedbacks and 4.7M multi-typed instructions. Specifically, to ensure MLLMs can handle diverse creativity-related queries, we prompt GPT to refine these human feedbacks to activate stronger creativity assessment capabilities. CreBench serves as a foundation for building MLLMs that understand human-aligned creativity. Based on the CreBench, we fine-tune open-source general MLLMs, resulting in CreExpert, a multimodal creativity evaluation expert model. Extensive experiments demonstrate that the proposed CreExpert models achieve significantly better alignment with human creativity evaluation compared to state-of-the-art MLLMs, including the most advanced GPT-4V and Gemini-Pro-Vision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。