评测大模型从文本提取结构化数据的能力，发现提示策略比模型大小更重要。
LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
- 构建涵盖22个模型、5种提示策略的多场景验证集
- 小模型用好提示可保证输出结构正确，但可能增加语义错误
- 引入分层评估指标，兼顾字符级准确率与文档级有效性
我们提出LLMStructBench，一个用于评估大语言模型从自然语言文本中提取结构化数据并生成有效JSON输出的新基准。该开源数据集包含多样化、人工验证的解析场景，涵盖不同复杂度，支持对22个模型和5种提示策略的系统性测试。我们进一步引入互补性能指标，同时捕捉字符级准确率和文档级有效性，实现对模型规模、提示方式等影响解析可靠性的严谨比较。结果显示，选择合适的提示策略比模型大小更具决定性作用：尤其对小模型或可靠性较差的模型，合理提示能显著提升输出结构有效性，但可能伴随语义错误增加。本基准为未来大模型在解析或数据抽取转换加载（ETL）应用中的研究提供重要支撑。
原文摘要 · Abstract (English)
We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。