arXiv:2602.00307cs.AIcs.DB2026-02

用元智能体自动构建优化数据流水线,适应变化需求。

Autonomous Data Processing using Meta-Agents

  • 通过分层智能体架构动态设计数据处理流程
  • 支持持续监控与迭代优化,提升流水线自适应能力
  • 适合需要自动化运维的数据工程团队

传统数据处理流水线多为静态手写,难以应对需求变化。尽管通用智能体和编程助手可生成明确任务的代码,但缺乏部署后自主监控、管理与优化的能力。我们提出自主数据处理框架ADP-MA,通过层级化智能体编排,实现数据流水线的动态构建、执行与迭代优化。核心为元智能体,可根据输入数据与任务要求设计多阶段计划,调用专用底层智能体,并持续评估流水线性能。系统包含三个关键组件:策略生成的规划模块、智能体协同与工具集成的编排层,以及用于迭代评估与回溯的监控环。相较于传统方法,ADP-MA强调上下文感知优化、自适应负载划分与渐进采样以增强可扩展性。同时,框架支持多种外部工具调用及已有智能体复用,减少冗余并加速构建。通过交互式演示,验证了其在典型数据处理任务中从流水线构建、执行监控到自适应优化的全流程能力。

原文摘要 · Abstract (English)

Traditional data processing pipelines are typically static and handcrafted for specific tasks, limiting their adaptability to evolving requirements. While general-purpose agents and coding assistants can generate code for well-understood data pipelines, they lack the ability to autonomously monitor, manage, and optimize an end-to-end pipeline once deployed. We present \textbf{Autonomous Data Processing using Meta-agents} (ADP-MA), a framework that dynamically constructs, executes, and iteratively refines data processing pipelines through hierarchical agent orchestration. At its core, \textit{meta-agents} analyze input data and task specifications to design a multi-phase plan, instantiate specialized \textit{ground-level agents}, and continuously evaluate pipeline performance. The architecture comprises three key components: a planning module for strategy generation, an orchestration layer for agent coordination and tool integration, and a monitoring loop for iterative evaluation and backtracking. Unlike conventional approaches, ADP-MA emphasizes context-aware optimization, adaptive workload partitioning, and progressive sampling for scalability. Additionally, the framework leverages a diverse set of external tools and can reuse previously designed agents, reducing redundancy and accelerating pipeline construction. We demonstrate ADP-MA through an interactive demo that showcases pipeline construction, execution monitoring, and adaptive refinement across representative data processing tasks.

智能体数据流水线自动化自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。