arXiv:2607.20465cs.LGcs.CL2026-07被引 1

首个评估大模型数据准备能力的基准,涵盖数据构建与质量评估。

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

论文配图:DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
图 1 · 摘自论文原文
  • 用统一下游任务衡量数据构建能力,基于原始数据生成训练集。
  • 新指标DAS在数学、科学、医疗领域相关性超0.7,领先现有方法。
  • 适合研究数据流水线、自动化训练数据生成的开发者和研究员。

训练数据质量从根本上决定大语言模型的能力,但目前缺乏统一的基准来评估大模型、智能体及数据驱动工作流在端到端数据准备中的表现。本文将大模型驱动的数据准备视为两个互补能力:数据构建,即从原始资料生成有监督训练数据;数据质量评估,即在下游训练前预测候选数据集的训练价值,其中“质量”指下游训练效用而非表面文本特征。我们提出DataPrep-Bench,首个统一基准,基于六类领域和多个基础模型,在共享下游任务协议下联合评估这两项能力。在数据构建方面,各方法使用相同原始数据源,评分方式为在输出数据上微调基础模型并联合使用Dolly-15k进行评估;同时发布技能引导智能体Data-Construction-Skill,相比仅使用Dolly的基线,在Llama-3.1-8B金融领域提升近20分绝对值,且在知识密集型领域表现媲美最强的智能体与DataFlow方法。在数据质量评估方面,评分函数通过与下游性能的皮尔逊相关系数评估,我们发布分布对齐得分(DAS),利用候选数据集与领域代理之间的最大均值差异(MMD)进行评估;DAS在六个领域中有四个达到最强跨模型相关性,且是唯一在数学、科学、医疗三个领域同时实现r > 0.70的指标,优于现有基于质量、多样性或启发式的方法。DataPrep-Bench提供了一个统一、以下游任务为基准的框架,将两项能力视为同等重要的目标,推动大模型驱动数据准备的发展。

原文摘要 · Abstract (English)

The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.

数据构建质量评估基准测试大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。