arXiv:2608.23719cs.CL2026-08中稿 · EMNLP

用闭环迭代提升生成数据质量,让大模型更懂人类真实需求。

ADE: Agentic Data Evolution Framework for Human-Centered Objectives

论文配图:ADE: Agentic Data Evolution Framework for Human-Centered Objectives
图 1 · 摘自论文原文
  • 通过观察-变异-选择闭环,动态优化合成数据
  • 在DEV300上内在成功率从50%升至75.81%
  • 适合需要可靠人工对齐的教育类任务

当目标不可执行且依赖上下文时,大语言模型难以对齐人类中心目标,限制了可验证性和可扩展监督。尽管合成数据拓展了覆盖范围,但弱验证将瓶颈从生成转移至筛选。噪声信号会破坏迭代优化,引发隐性退化。我们提出代理数据演化框架(ADE),将合成监督组织为持续演化的数据快照。ADE通过观察-变异-选择(OVS)闭环改进数据快照,稳态准入机制如质量棘轮,保守地控制更新以实现跨轮次持续提升。通过内在趋势追踪与外在训练后评估验证改进效果。在DEV300上,内在胜率从50%提升至75.81%,外在胜率从55.20%提升至68.86%,各类基准表现一致提升。盲评专家偏好率达66.11%。性能增益适用于多种后训练方法、模型规模及超出目标弱可验证教育任务的场景。资源见https://github.com/ZeroLoss-Lab/Agentic-Data-Evolution。

原文摘要 · Abstract (English)

Aligning large language models to human-centered objectives is difficult when targets are non-executable and context-dependent, limiting reliable verification and scalable supervision. Although synthetic data expands coverage, weak verification shifts the bottleneck from generation to selection. Noisy signals destabilize iterative refinement and can cause silent regressions. We propose Agentic Data Evolution (ADE), a data-centric framework that organizes synthetic supervision as evolving data snapshots. ADE improves data snapshots through a closed-loop Observation-Variation-Selection (OVS) procedure, where a steady-state admission mechanism acts as a quality ratchet that conservatively gates updates for sustained cross-round improvement. We validate these improvements through complementary intrinsic trend tracking and extrinsic post-training evaluation. On DEV300, ADE raises the intrinsic win rate from 50% to 75.81% and the extrinsic win rate from 55.20% to 68.86%, consistent performance gains across diverse benchmarks. Blind expert evaluation further confirms this, with a 66.11% preference for evolved answers. These gains extend across post-training methods, model scales, and tasks beyond the target weakly verifiable educational objectives. Resources are available at https://github.com/ZeroLoss-Lab/Agentic-Data-Evolution.

数据演化人类对齐大模型训练合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。