用大模型自动生成数据契约，减少70%人工工作量
AI-Driven Generation of Data Contracts in Modern Data Engineering Systems
- 用LoRA等方法微调大模型，根据数据样例生成契约
- 在真实和合成数据上生成契约准确率高，人工工作量降超70%
- 适合需要快速迭代数据治理的企业数据团队
数据契约定义了数据生产者与消费者之间关于模式、语义和质量的协议。随着数据管道日益复杂，手动编写和维护契约变得易出错且耗时。本文提出一种基于大语言模型（LLMs）的AI驱动框架，实现数据契约的自动生成。系统采用参数高效微调技术（如LoRA、PEFT），使大模型适应结构化数据领域。模型输入数据样本或模式描述，输出符合JSON Schema、Avro等格式的已验证契约定义。该框架已集成至现代数据平台（如Databricks、Snowflake），支持大规模契约执行。在合成与真实数据集上的实验表明，微调后的模型能高效生成有效契约，人工工作量降低超过70%。同时讨论了幻觉、版本控制及持续学习等关键挑战。本研究证明生成式AI可实现可扩展、敏捷的数据治理，弥合企业数据管理中意图与实现之间的鸿沟。
原文摘要 · Abstract (English)
Data contracts formalize agreements between data producers and consumers regarding schema, semantics, and quality expectations. As data pipelines grow in complexity, manual authoring and maintenance of contracts becomes error-prone and labor-intensive. We present an AI-driven framework for automatic data contract generation using large language models (LLMs). Our system leverages parameter-efficient fine-tuning methods, including LoRA and PEFT, to adapt LLMs to structured data domains. The models take sample data or schema descriptions and output validated contract definitions in formats such as JSON Schema and Avro. We integrate this framework into modern data platforms (e.g., Databricks, Snowflake) to automate contract enforcement at scale. Experimental results on synthetic and real-world datasets demonstrate that the fine-tuned LLMs achieve high accuracy in generating valid contracts and reduce manual workload by over 70%. We also discuss key challenges such as hallucination, version control, and the need for continuous learning. This work demonstrates that generative AI can enable scalable, agile data governance by bridging the gap between intent and implementation in enterprise data management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。