评测大模型生成结构化输出的能力，涵盖文本与可视化格式。
StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
- 设计两类任务：自然语言生成结构数据，以及格式间转换。
- 顶尖模型平均得分75.58，开源模型落后约10分。
- 生成视觉内容比纯文本结构更难，生成任务整体更挑战。
随着大语言模型（LLMs）日益融入软件开发流程，其生成结构化输出的能力变得至关重要。我们提出 StructEval，一个全面的基准测试，用于评估 LLM 在生成非渲染类（如 JSON、YAML、CSV）和可渲染类（如 HTML、React、SVG）结构化格式方面的能力。不同于以往基准，StructEval 通过两种范式系统性评估结构保真度：1）生成任务，从自然语言提示生成结构化输出；2）转换任务，实现不同结构化格式间的相互转换。该基准涵盖 18 种格式与 44 类任务，并引入新型指标衡量格式遵循度与结构正确性。结果表明存在显著性能差距——即使是表现最优的 o1-mini 模型，平均得分也仅达 75.58，而开源模型则落后约 10 分。我们发现生成任务比转换任务更具挑战性，生成正确视觉内容比生成纯文本结构更困难。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and \textbf{2)} conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。