让大模型自动优化数据结构,提升信息抽取准确率与可靠性
PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction
- 用大模型动态优化JSON schema,使其更适配自身理解
- 在多个数据集上提升抽取准确率最高达64.7%,错误减少92%
- 适合需要高可靠性的自动化系统与智能代理开发人员
从非结构化文本中提取结构化信息对新兴的Software 3.0系统至关重要,其中大模型代理需自主调用API与工具。现有方法直接使用静态JSON schema进行抽取,常依赖约束解码或强化学习保证语法正确性,但将schema视为为人类开发者设计的固定契约,导致性能不佳、幻觉频发及代理行为不可靠。我们发现JSON schema本身是自然语言理解的契约,可被大模型解读并改进。因此提出PARSE(Parameter Automated Refinement and Schema Extraction),包含两个协同组件:ARCHITECT自动优化JSON schema以适配大模型,通过RELAY(集成代码生成系统)保持向后兼容;SCOPE实现基于反思的抽取,结合静态与大模型双重校验机制。在Schema-Guided Dialogue (SGD)、Structured Web Data Extraction (SWDE)及内部零售对话数据集上评估,结果显示在SWDE上抽取准确率最高提升64.7%,综合框架改进使跨模型准确率提升10%,首次重试中错误减少92%,且保持实用延迟。
原文摘要 · Abstract (English)
Structured information extraction from unstructured text is critical for emerging Software 3.0 systems where LLM agents autonomously interact with APIs and tools. Recent approaches apply large language models directly to extraction tasks using existing JSON schemas, often with constraint decoding or reinforcement learning approaches to ensure syntactic validity, but treat JSON schemas as static contracts designed for human developers, leading to suboptimal extraction performance, frequent hallucinations, and unreliable agent behavior when schemas contain ambiguous or incomplete specifications. We recognize that JSON schemas themselves are a form of natural language understanding contract that encodes rules, relationships, and expectations about data structure contracts that LLMs should be able to both interpret and systematically improve. Consequently, we develop PARSE (Parameter Automated Refinement and Schema Extraction), a novel system with two synergistic components: ARCHITECT, which autonomously optimizes JSON schemas for LLM consumption while maintaining backward compatibility through RELAY (an integrated code generation system), and SCOPE, which implements reflection-based extraction with combined static and LLM-based guardrails. We evaluate PARSE qualitatively and quantitatively on three datasets including Schema-Guided Dialogue (SGD), Structured Web Data Extraction (SWDE), and internal retail conversation data, and find that it achieves up to 64.7% improvement in extraction accuracy on SWDE with combined framework improvements reaching 10% across models, while reducing extraction errors by 92% within the first retry and and maintaining practical latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。