用表格自动生成自然语言文本,构建高质量抽取评测基准。
StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation
- 基于已有表格数据,分两阶段生成对应自然语言文本。
- 在49个数据集上验证,模型事实准确但叙事连贯性差。
- 提供多维度评估工具和开源框架,支持持续研究。
从文本中提取结构化信息(如键值对)在企业应用中极具价值。尽管大语言模型(LLMs)已实现多项自然语言到结构化格式的自动化转换,但在特定领域或组织文档中仍缺乏评测基准。手动标注成本高,难以扩展。本文提出StructText,一个端到端框架,利用现有表格数据作为结构化真实标签,通过“先规划后执行”的两阶段流程合成对应的自然语言文本。为确保文本与结构数据一致,引入多维度评估策略:结合大模型判断事实性、幻觉与连贯性,以及客观的数值和时间准确性提取指标。在49个数据集共71,539个样本上评估发现,虽然模型事实准确且极少幻觉,但在生成可抽取文本的叙事连贯性方面表现不佳;数值与时间信息虽保真嵌入,却难以被自动提取。研究开源了框架、数据集、评估工具和基线系统以支持后续研究。
原文摘要 · Abstract (English)
Extracting structured information from text, such as key-value pairs that could augment tabular data, is quite useful in many enterprise use cases. Although large language models (LLMs) have enabled numerous automated pipelines for converting natural language into structured formats, there is still a lack of benchmarks for evaluating their extraction quality, especially in specific domains or focused documents specific to a given organization. Building such benchmarks by manual annotations is labour-intensive and limits the size and scalability of the benchmarks. In this work, we present StructText, an end-to-end framework for automatically generating high-fidelity benchmarks for key-value extraction from text using existing tabular data. It uses available tabular data as structured ground truth, and follows a two-stage ``plan-then-execute'' pipeline to synthetically generate corresponding natural-language text. To ensure alignment between text and structured source, we introduce a multi-dimensional evaluation strategy that combines (a) LLM-based judgments on factuality, hallucination, and coherence and (b) objective extraction metrics measuring numeric and temporal accuracy. We evaluated the proposed method on 71,539 examples across 49 datasets. Results reveal that while LLMs achieve strong factual accuracy and avoid hallucination, they struggle with narrative coherence in producing extractable text. Notably, models presume numerical and temporal information with high fidelity yet this information becomes embedded in narratives that resist automated extraction. We release a framework, including datasets, evaluation tools, and baseline extraction systems, to support continued research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。