提出新评估框架,精准衡量大模型生成结构化数据的一致性。
STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability
- 用语义树编辑距离衡量输出相似性,兼顾语义灵活与结构严格。
- 在合成数据上验证,新指标对语义等价输出相似度达0.86-0.90。
- 可帮助选型模型、调优提示词,适合生产级结构化输出场景。
大语言模型在结构化数据生成中应用日益广泛,但输出一致性仍是生产场景的关键挑战。本文提出一个全面的评估与改进框架,包含:(1) 新颖的语义树编辑距离(STED),在比较JSON输出时平衡语义灵活性与结构严格性;(2) 通过多次生成的STED测量聚合,量化输出可靠性的一致性评分框架。在具有可控模式、表达方式和语义变化的合成数据集上系统实验表明,STED在语义等价输出上达到0.86–0.90的相似度,结构断裂时为0.0,优于现有指标如TED、BERTScore和DeepDiff。将该框架应用于六款主流大模型的基准测试显示显著差异:Claude-3.7-Sonnet在高温(T=0.9)下仍保持近乎完美的结构一致性,而Claude-3-Haiku与Nova-Pro则表现出明显退化,需精细调参。该框架支持模型选型、提示词迭代优化及不一致根源诊断,为基于大模型的生产系统提供理论基础与实用工具。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed for structured data generation, yet output consistency remains critical for production applications. We introduce a comprehensive framework for evaluating and improving consistency in LLM-generated structured outputs. Our approach combines: (1) STED (Semantic Tree Edit Distance), a novel similarity metric balancing semantic flexibility with structural strictness when comparing JSON outputs, and (2) a consistency scoring framework aggregating multiple STED measurements across repeated generations to quantify reliability. Through systematic experiments on synthetic datasets with controlled schema, expression, and semantic variations, we demonstrate STED achieves superior performance ($0.86-0.90$ similarity for semantic equivalents, $0.0$ for structural breaks) compared to existing metrics including TED, BERTScore, and DeepDiff. Applying our framework to benchmark six LLMs reveals significant variations: Claude-3.7-Sonnet demonstrates exceptional consistency, maintaining near-perfect structural reliability even at high temperatures ($T=0.9$), while models like Claude-3-Haiku and Nova-Pro exhibit substantial degradation requiring careful tuning. Our framework enables practical applications including targeted model selection for structured tasks, iterative prompt refinement for reproducible results, and diagnostic analysis to identify inconsistency root causes. This work provides theoretical foundations and practical tools for ensuring reliable structured output generation in LLM-based production systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。