用强化学习自动生成适配大模型的数据配方，效果媲美人工。
DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning
- 通过强化学习在线优化数据处理流程，自动生成训练配方。
- 在6个任务上表现接近人工设计，数学领域达66.7分超越基线。
- 适合想自动化训练数据构建的研究者与工程师。
当前大语言模型的性能高度依赖大规模高质量训练数据。关键在于数据配方——将原始数据转化为训练语料的处理流程。尽管已有模型用于自动化数据合成与筛选等步骤，但整体配方设计仍依赖人工，耗时且需丰富经验。本文提出端到端数据配方生成方法，给定目标任务和数据源池，模型可输出适配该任务的完整数据配方。我们提出DataChef-32B，利用代理奖励函数预测下游性能，在线进行强化学习。在六个预留任务中，其生成的配方性能接近人类专家水平。尤其在数学领域，它使Qwen3-1.7B-Base模型在AIME'25上达到66.7分，超过官方微调版本（Qwen3-1.7B）。本工作为自动化大模型训练与自演化AI系统提供了新思路。
原文摘要 · Abstract (English)
In the current landscape of Large Language Models (LLMs), the curation of large-scale, high-quality training data is a primary driver of model performance. A key lever is the \emph{data recipe}, which comprises a data processing pipeline to transform raw sources into training corpora. Despite the growing use of LLMs to automate individual data processing steps, such as data synthesis and filtering, the overall design of data recipes remains largely manual and labor-intensive, requiring substantial human expertise and iteration. To bridge this gap, we formulate \emph{end-to-end data recipe generation} for LLM adaptation. Given a target benchmark and a pool of available data sources, a model is required to output a complete data recipe that adapts a base LLM to the target task. We present DataChef-32B, which performs online reinforcement learning using a proxy reward that predicts downstream performance for candidate recipes. Across six held-out tasks, DataChef-32B produces recipes that yield performance comparable to those curated by human experts. Notably, the recipe from DataChef-32B adapts Qwen3-1.7B-Base to the math domain, achieving 66.7 on AIME'25 and surpassing the official post-training checkpoint (Qwen3-1.7B). This work sheds new light on automating LLM training and developing self-evolving AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。