arXiv:2603.20311cs.SEcs.AI2026-03被引 1

用自然语言自动生成可运行的数据流水线,解决意图模糊和出错问题。

kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation

  • 通过追问澄清用户意图,再生成任务专用的转换组件。
  • 提取加载成功率提升3倍,转换准确率提高25%。
  • 适合需要快速搭建数据管道的研发团队使用。

现代机器学习系统依赖复杂的数据工程流程来完成数据的抽取、转换和加载(ELT)。然而,构建这些流水线耗时且需要深厚的数据基础设施与编排框架知识。大语言模型(LLM)代理虽为自动化提供了可能,但现有方法在用户意图不明确、工具生成不可靠及输出不可执行方面仍存在局限。我们提出kRAIG,一个将自然语言描述转化为生产就绪的Kubeflow Pipelines(KFP)的AI代理。为解决意图歧义,提出ReQuesAct(Reason, Question, Act)交互框架,在生成流水线前主动澄清意图。系统从多种数据源端到端调度数据流动,并通过检索增强的工具合成过程生成特定任务的转换组件。为保障数据质量和安全,kRAIG引入基于LLM的验证阶段,执行前检查流水线完整性。相比最先进代理基线,该框架在数据提取与加载成功率上提升3倍,在转换准确率上提高25%。结果表明,带有显式意图澄清与验证的结构化代理工作流能显著提升自动化数据工程流水线的可靠性与可执行性。

原文摘要 · Abstract (English)

Modern machine learning systems rely on complex data engineering workflows to extract, transform, and load (ELT) data into production pipelines. However, constructing these pipelines remains time-consuming and requires substantial expertise in data infrastructure and orchestration frameworks. Recent advances in large language model (LLM) agents offer a potential path toward automating these workflows, but existing approaches struggle with under-specified user intent, unreliable tool generation, and limited guarantees of executable outputs. We introduce kRAIG, an AI agent that translates natural language specifications into production-ready Kubeflow Pipelines (KFP). To resolve ambiguity in user intent, we propose ReQuesAct (Reason, Question, Act), an interaction framework that explicitly clarifies intent prior to pipeline synthesis. The system orchestrates end-to-end data movement from diverse sources and generates task-specific transformation components through a retrieval-augmented tool synthesis process. To ensure data quality and safety, kRAIG incorporates LLM-based validation stages that verify pipeline integrity prior to execution. Our framework achieves a 3x improvement in extraction and loading success and a 25 percent increase in transformation accuracy compared to state-of-the-art agentic baselines. These improvements demonstrate that structured agent workflows with explicit intent clarification and validation significantly enhance the reliability and executability of automated data engineering pipelines.

数据管道LLM代理自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。