评测大模型在真实数据治理任务中的表现,提出新框架显著提升准确率与调试效率。
DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
- 设计反向目标法生成真实噪声数据,构建150个实际场景任务
- 现有模型在复杂流程中准确率仅39.7,新框架提升至54.9
- 适合关注数据治理自动化、AI可信落地的研究者与工程师
数据治理通过政策与标准保障数据质量、安全与合规性,是现代AI发展的关键基础。近期大语言模型(LLMs)被用于将用户意图转化为可执行的数据转换代码,实现自动化。然而,现有基准多聚焦代码片段或高层分析,未能反映数据治理的核心挑战:确保数据本身正确与高质量。为此,我们提出DataGovBench,一个包含150个基于真实案例的多样化任务的基准,采用新颖的“反向目标”方法合成真实噪声,并使用严格指标评估端到端流程可靠性。分析显示,当前模型在复杂多步流程中表现不佳,缺乏稳健纠错能力。因此,我们提出DataGovAgent框架,采用规划-执行-评估架构,融合约束规划、检索增强生成与沙箱反馈调试。实验表明,该框架将复杂任务平均任务得分(ATS)从39.7提升至54.9,调试迭代次数减少超过77.9%。
原文摘要 · Abstract (English)
Data governance ensures data quality, security, and compliance through policies and standards, a critical foundation for scaling modern AI development. Recently, large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into executable transformation code. However, existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics, failing to capture the unique challenge of data governance: ensuring the correctness and quality of the data itself. To bridge this gap, we introduce DataGovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases. DataGovBench employs a novel "reversed-objective" methodology to synthesize realistic noise and utilizes rigorous metrics to assess end-to-end pipeline reliability. Our analysis on DataGovBench reveals that current models struggle with complex, multi-step workflows and lack robust error-correction mechanisms. Consequently, we propose DataGovAgent, a framework utilizing a Planner-Executor-Evaluator architecture that integrates constraint-based planning, retrieval-augmented generation, and sandboxed feedback-driven debugging. Experimental results show that DataGovAgent significantly boosts the Average Task Score (ATS) on complex tasks from 39.7 to 54.9 and reduces debugging iterations by over 77.9 percent compared to general-purpose baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。