用分步分析与判断分离法,提升大模型对复杂创意任务的评分可靠性。
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

- 将评价拆分为结构化分析和基于证据判断两阶段,减少偏见干扰。
- 在复杂创意任务上比最优基线平均提升22.74%的评估准确率。
- 适合需要高可信度创意评分的研究者与教育平台使用。
自动化创意任务评估对大语言模型作为评判者仍具挑战性,因模型易受冗长偏差和宽松偏差影响。此类局限在上下文关联且流程结构化的多步骤创意任务(CGPST)中尤为显著,其步骤间依赖性强、主观性高、评分范围广,导致判断更不稳定且易偏倚。现有方法或依赖特定任务训练,或直接应用大模型作为评判者,均难以在复杂场景下保证评估可靠性。为此,我们提出CreaEval,一种面向CGPST的自动化创意评估框架,将传统的大模型评判解耦为分析与判断两个阶段。具体包含:记忆增强型分析阶段,通过思维链大模型将多步响应转化为结构化评估证据,并引入跨步骤记忆;基于证据的判断阶段,评判大模型仅依据提取的证据进行打分,不接触原始响应。综合实验表明,CreaEval在CGPST及两项经典简单创意任务上,平均性能优于次优基线22.74%,展现出良好泛化能力。代码已开源:https://github.com/Jaong/CreaEval。
原文摘要 · Abstract (English)
Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose CreaEval, an automated creativity evaluator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: Memory-augmented Analysis, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and Evidence-based Judging, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments show that CreaEval achieves an average performance improvement of 22.74% over the second-best baselines across CGPST and two classic simple creativity tasks, demonstrating its generalizability. The code is available at https://github.com/Jaong/CreaEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。