用大模型自动构建可复现的数据流水线,提升训练数据质量与效率。
DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI
- 基于大模型自动生成数据处理流程,支持模块化组合与调试优化。
- 在数学、代码等任务上性能超人工数据集,最高提升7%执行准确率。
- 适合需要高质量数据的AI研发团队,尤其擅长自动化数据生成与处理。
大型语言模型对高质量数据的需求日益增长,但现有数据准备流程多依赖临时脚本和非规范工作流,缺乏可复现性与模型协同能力。为此,我们提出DataFlow——一个统一且可扩展的LLM驱动数据准备框架。该框架采用系统级抽象,支持模块化、可重用、可组合的数据变换,并提供类似PyTorch的流水线构建API,实现可调试、可优化的数据流。框架包含近200个可复用算子及六类通用数据流水线,覆盖文本、数学推理、代码生成、Text-to-SQL、智能代理RAG和大规模知识抽取。为进一步提升易用性,引入DataFlow-Agent,通过算子合成与迭代验证,将自然语言需求自动转化为可执行流水线。在六个典型场景中,DataFlow持续提升下游模型性能:数学、代码和文本流水线优于人工标注数据集与专用合成基线,在Text-to-SQL任务上比SynSQL高+3%执行准确率,代码基准平均提升+7%,在MATH、GSM8K和AIME上提升1–3分。此外,由DataFlow生成的10,000样本统一数据集,使基础模型性能超越使用100万条Infinity-Instruct数据训练的模型。结果表明,DataFlow为可靠、可复现、可扩展的LLM数据准备提供了高效技术底座,奠定了数据驱动型AI发展的系统基础。
原文摘要 · Abstract (English)
The rapidly growing demand for high-quality data in Large Language Models (LLMs) has intensified the need for scalable, reliable, and semantically rich data preparation pipelines. However, current practices remain dominated by ad-hoc scripts and loosely specified workflows, which lack principled abstractions, hinder reproducibility, and offer limited support for model-in-the-loop data generation. To address these challenges, we present DataFlow, a unified and extensible LLM-driven data preparation framework. DataFlow is designed with system-level abstractions that enable modular, reusable, and composable data transformations, and provides a PyTorch-style pipeline construction API for building debuggable and optimizable dataflows. The framework consists of nearly 200 reusable operators and six domain-general pipelines spanning text, mathematical reasoning, code, Text-to-SQL, agentic RAG, and large-scale knowledge extraction. To further improve usability, we introduce DataFlow-Agent, which automatically translates natural-language specifications into executable pipelines via operator synthesis, pipeline planning, and iterative verification. Across six representative use cases, DataFlow consistently improves downstream LLM performance. Our math, code, and text pipelines outperform curated human datasets and specialized synthetic baselines, achieving up to +3\% execution accuracy in Text-to-SQL over SynSQL, +7\% average improvements on code benchmarks, and 1--3 point gains on MATH, GSM8K, and AIME. Moreover, a unified 10K-sample dataset produced by DataFlow enables base models to surpass counterparts trained on 1M Infinity-Instruct data. These results demonstrate that DataFlow provides a practical and high-performance substrate for reliable, reproducible, and scalable LLM data preparation, and establishes a system-level foundation for future data-centric AI development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。