新基准QUIET用多空连环故事测试大模型创造力，自动评分更客观。
QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability
- 设计多空连环故事，前后空格有依赖约束，逼出真实创作能力。
- 用信息论公式评分：满足约束+出人意料=高分，零分淘汰不合格答案。
- 无需人工打分，适合评估模型在开放生成中的创意水平。
大语言模型在创意能力评估上面临双重挑战：现有基准（如Story Cloze Test、HellaSwag）采用多选识别范式，衡量的是模型对续写内容的判别能力，而非直接生成能力；基于评分标准或LLM作为裁判的方法依赖主观判断或模型输出，缺乏客观自动化评分机制。本文提出QUIET（Quality Understanding via Interlocked Evaluation Testing），一种基于多空连环故事闭塞的诊断性基准，用于评估大模型的创造性能力。QUIET在结构完整的故事中设置10-20个空白，每个空白配有明确的内容约束，并建立空格间的级联依赖关系——早期填空内容会限制后续填空的可行解空间。被评估模型以开放式生成方式填写所有空白，结果通过信息论驱动的自动化评分协议进行评分，无需人工干预。该评分协议直接实现“校准惊喜”理论框架（Zou & Xu, 2026a）。对每个空白k，复合得分计算为：score = satisfy * (1 + lambda * surprise)，其中lambda = 1.0。'satisfy'衡量答案满足内容约束的程度（基于客观逻辑推理判断，非主观审美评价），'surprise'衡量在满足约束前提下的意外程度。不满足约束的答案得分为零；满足但平庸的答案得分低；满足且出人意料的答案得分高。
原文摘要 · Abstract (English)
Large language models (LLMs) face a dual challenge in creative capability evaluation: existing benchmarks (e.g., Story Cloze Test, HellaSwag) measure models' discriminative ability over narrative continuation using multiple-choice recognition paradigms, rather than directly measuring creative generation capability; rubric-based scoring and LLM-as-Judge methods rely on subjective dimension assessment or natural language model outputs, and cannot provide objective, automated scoring mechanisms. This paper proposes QUIET (Quality Understanding via Interlocked Evaluation Testing), a diagnostic benchmark for LLM creative capability based on multi-blank cascaded story cloze. QUIET sets N blanks (10-20) in a story with complete structure, with each blank accompanied by an explicit content constraint, and cascade dependency relationships between blanks -- the content filled into earlier blanks constrains the feasible solution space for later blanks. The evaluated model (or human participants) fills all blanks in open-ended generation mode; the results are scored by an information-theoretic automated scoring protocol without human grading. The scoring protocol directly operationalizes the "calibrated surprise" theoretical framework (Zou & Xu, 2026a). For each blank k, a composite score is computed: score = satisfy * (1 + lambda * surprise), where lambda = 1.0. Here, "satisfy" measures how well the blank filling satisfies the content constraint (objective logical reasoning judgment, not subjective aesthetic scoring), and "surprise" measures the degree of surprise given that the constraint is satisfied. Creative answers that do not satisfy the constraint score zero; answers that satisfy the constraint but are mediocre score low; answers that satisfy the constraint and are surprising score high.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。