评测大模型结构化输出能力，跨文本、图像、音频三类数据源。
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
- 构建多源数据集，统一文本表示以隔离模态处理差异。
- 图像和音频抽取准确率分别仅67.2%和23.7%，长上下文更难。
- 适合评估多模态结构化生成模型，关注真实场景应用。
大语言模型日益用于从非结构化或半结构化数据中提取结构化信息，如发票、病历及将PDF转为数据库条目。然而现有基准要么仅关注模式合规性，要么局限于单一数据源的价值正确性。我们提出SOB（Structured Output Benchmark），涵盖文本、图像和音频三类模态的多源基准。所有模型接收统一的文本化上下文表示，避免原始视觉或语音处理能力干扰，实现源无关的公平比较。基准包含5,000条文本样本（源自25,091条全语料的多跳问答）、209张图像样本（来自七种文档类型的OCR PDF，含多栏布局、密集表格、历史扫描件、小字文本与数学排版）以及115段音频样本（来自AMI语料库）。每条记录包含自然语言问题、需遵循的JSON模式及基于源上下文验证的真值答案。我们在三个数据源上评估21个前沿与开源模型，使用七项指标。结果表明：模型在模式合规性上接近完美，但最佳值准确率（精确叶子值匹配）在文本上为83.0%，图像上为67.2%，音频上仅为23.7%，且上下文越长，抽取难度越大。数据集、评估流程与代码均已开源。
原文摘要 · Abstract (English)
Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。