评测大模型生成结构化数据的能力，发现主流工具存在效率与兼容性短板。
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- 构建1万条真实JSON Schema的评测基准，覆盖多种复杂约束。
- 六款主流工具在覆盖率和生成质量上差异显著，部分无法处理复杂嵌套结构。
- 适合开发者优化结构化输出系统，也适用于评估模型生成可靠性。
可靠生成结构化输出已成为现代语言模型应用的关键能力。约束解码已成为各领域生成过程中确保结构一致性的主流技术。尽管广泛应用，但对约束解码行为与性能的系统性评估仍严重不足。当前约束解码框架普遍以JSON Schema作为结构化数据格式标准，多数声称能保证在给定模式下生成合规输出。然而，实际效果尚不清晰。本文提出一个评估框架，从生成效率、约束类型覆盖范围、输出质量三个维度评估约束解码方法。为此，我们构建了JSONSchemaBench——一个包含10,000个真实世界JSON Schema的基准测试集，涵盖广泛且复杂度多样的约束。结合官方JSON Schema测试套件，我们评估了六种前沿约束解码框架：Guidance、Outlines、Llamacpp、XGrammar、OpenAI和Gemini。通过大量实验，揭示了约束解码在真实场景下的能力边界与局限。本研究为改进约束解码框架与结构化生成任务提供了可操作的洞见，确立了新的评估标准。代码与数据已开源：https://github.com/guidance-ai/jsonschemabench。
原文摘要 · Abstract (English)
Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology across sectors for enforcing structured outputs during generation. Despite its growing adoption, little has been done with the systematic evaluation of the behaviors and performance of constrained decoding. Constrained decoding frameworks have standardized around JSON Schema as a structured data format, with most uses guaranteeing constraint compliance given a schema. However, there is poor understanding of the effectiveness of the methods in practice. We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs. To facilitate this evaluation, we introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity. We pair the benchmark with the existing official JSON Schema Test Suite and evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini. Through extensive experiments, we gain insights into the capabilities and limitations of constrained decoding on structured generation with real-world JSON schemas. Our work provides actionable insights for improving constrained decoding frameworks and structured generation tasks, setting a new standard for evaluating constrained decoding and structured generation. We release JSONSchemaBench at https://github.com/guidance-ai/jsonschemabench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。