用强化学习自动优化生成高质量合成数据的流程,无需人工标注。
AutoSynth: Automated Workflow Optimization for High-Quality Synthetic Dataset Generation via Monte Carlo Tree Search
- 将数据生成流程优化转化为无参考数据的蒙特卡洛树搜索。
- 在教育类主观任务中,生成数据训练的模型性能超基线40-51%。
- 只需30分钟即可完成,人力投入减少90%以上,适合资源有限场景。
大语言模型在特定任务上的监督微调需要高质量数据集,但人工标注成本过高。合成数据生成具备可扩展性,但其效果依赖复杂的多阶段工作流,涉及提示工程与模型编排。现有自动化方法存在冷启动问题:需标注数据进行奖励建模,这对无客观真值的主观开放任务尤为不利。我们提出AutoSynth,一种无需参考数据的框架,通过将问题重构为基于新型无数据混合奖励的蒙特卡洛树搜索来实现工作流发现与优化。该奖励通过两个大模型作为裁判组件实现元学习:一个利用动态生成的任务特定指标评估样本质量,另一个评估工作流代码与提示质量。在主观教育任务上的实验表明,尽管专家设计的工作流在人类偏好上胜出(96-99%胜率),但使用AutoSynth生成数据训练的模型显著优于基线(40-51%对比2-5%),并在某些指标上达到或超越专家水平,暗示发现超出人类直觉的质量维度。整个过程将人工耗时从5-7小时降至30分钟(降幅>90%)。AutoSynth解决了数据驱动AI中的冷启动问题,为主观大模型任务提供了一种可扩展、低成本的解决方案。代码见:https://github.com/bisz9918-maker/AutoSynth。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) of large language models (LLMs) for specialized tasks requires high-quality datasets, but manual curation is prohibitively expensive. Synthetic data generation offers scalability, but its effectiveness relies on complex, multi-stage workflows, integrating prompt engineering and model orchestration. Existing automated workflow methods face a cold start problem: they require labeled datasets for reward modeling, which is especially problematic for subjective, open-ended tasks with no objective ground truth. We introduce AutoSynth, a framework that automates workflow discovery and optimization without reference datasets by reframing the problem as a Monte Carlo Tree Search guided by a novel dataset-free hybrid reward. This reward enables meta-learning through two LLM-as-judge components: one evaluates sample quality using dynamically generated task-specific metrics, and another assesses workflow code and prompt quality. Experiments on subjective educational tasks show that while expert-designed workflows achieve higher human preference rates (96-99% win rates vs. AutoSynth's 40-51%), models trained on AutoSynth-generated data dramatically outperform baselines (40-51% vs. 2-5%) and match or surpass expert workflows on certain metrics, suggesting discovery of quality dimensions beyond human intuition. These results are achieved while reducing human effort from 5-7 hours to just 30 minutes (>90% reduction). AutoSynth tackles the cold start issue in data-centric AI, offering a scalable, cost-effective method for subjective LLM tasks. Code: https://github.com/bisz9918-maker/AutoSynth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。