提出评估大模型智能体数据质量的ACE框架,解决生成数据冗余无效问题。
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents

- 将智能体数据分解为环境、任务、交互和验证器四要素,统一建模
- 用准确度、复杂度、多样性三维度设计生成数据分布,避免盲目堆量
- 揭示当前趋势:数据需贴合执行场景、匹配学习者能力、覆盖真实行为
大语言模型智能体越来越依赖生成的交互数据来学习与外部环境互动。智能体数据生成必须在环境、任务、交互和成功信号之间保持一致性,同时产出有用而非仅仅是大量的经验。现有研究涵盖多个智能体领域,但以领域为中心的组织方式和异构评估常常掩盖了通用生成机制,并混淆了候选生成与验证选择。本文构建了一个双层框架:首先,将智能体数据表示为统一的因子化对象 (E, q, τ, v),包含环境规范、任务信号、交互实现和可选验证器;其次,通过准确度-复杂度-多样性(ACE)视角,将生成建模为受限分布设计。准确度确保数据具备实际可行且内部一致的支持域;在此支持域内,复杂度根据声明的学习者能力和执行配置分配学习权重,多样性则控制数据覆盖范围与冗余程度。基于此框架,我们分析了以往工作如何验证生成经验、构造与校准难度、扩展行为覆盖。文献揭示出向执行锚定的准确度、学习者相关的复杂度以及超越表面变化或数据集大小的多样性转变趋势。我们进一步通过ACE视角讨论了智能体数据生成的更广方向与新兴趋势,包括对规模化、数据来源、训练范式和自适应学习的影响。总体而言,核心挑战并非生成更多数据,而是在智能体与环境演进过程中持续分配有效、信息丰富且非冗余的经验。
原文摘要 · Abstract (English)
LLM agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object $(E,q,τ,v)$, comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as agents and environments evolve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。