构建统一评估框架,衡量大模型在收敛与发散任务中的创造力。
What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Models
- 区分收敛与发散创造力,基于社会科学研究设计评估维度。
- 在主流大模型上测试,揭示其在实用、新颖与意外性上的表现差异。
- 适合关注生成式AI创造力评估的研究者与开发者参考。
基础模型(FMs)的迅猛发展使其能力远超传统任务。创造力作为人类智能的标志和创新驱动力,正日益被视为生成式基础模型中机器智能的关键维度,补充了传统的准确性指标。然而,现有创造力评估框架仍零散,依赖未扎根于成熟理论的临时指标。为此,我们提出C^2-Eval,一个全面的基准,用于统一评估基础模型的创造力。C^2-Eval区分两种互补的创造力形式:收敛创造力(如代码生成,解有约束)与发散创造力(如故事创作,开放性任务)。它采用来自社会科学理论的细粒度标准,聚焦实用性(Usefulness)、原创性(Originality)与意外性(Surprise,简称U-O-S)。通过对领先专有与开源模型的广泛实验,我们分析了其创造力能力间的权衡。结果揭示了当前基础模型在追求创造性机器心智过程中的优势与挑战,表明C^2-Eval是审视创造性AI演进图景的有效工具。
原文摘要 · Abstract (English)
The meteoric rise of foundation models (FMs) has expanded their capabilities far beyond conventional tasks. Creativity, long regarded as a hallmark of human intelligence and a driver of innovation, is now increasingly recognized as a critical dimension of machine intelligence in the era of generative FMs, complementing traditional measures of accuracy. However, existing evaluation frameworks for creativity remain fragmented, relying on ad hoc metrics not firmly grounded in established theories. To address this gap, we introduce C^2-Eval, a holistic benchmark for unified assessment of creativity in FMs. C^2-Eval distinguishes between two complementary forms of creativity: convergent creativity, where tasks admit constrained solutions (e.g., code generation), and divergent creativity, where tasks are open-ended (e.g., storytelling). It evaluates both dimensions using fine-grained criteria derived from social-science theory, focusing on Usefulness, Originality, and Surprise (U-O-S). Through extensive experiments on leading proprietary and open-source models, we analyze trade-offs in their creative capabilities. Our results highlight both the strengths and challenges of current FMs in pursuing a creative machine mind, showing that C^2-Eval is an effective lens for examining the evolving landscape of creative AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。