用真实企业任务评估智能体自我进化能力,发现现有模型仍有巨大提升空间。
GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks
- 通过规则重组机制拆解企业流程,实现训练与测试任务间的可追溯进化
- 在12个业务领域中验证,自进化使准确率最高提升16.44个百分点
- 全自动数据流水线支持快速扩展,适合评估智能体长期学习能力
智能体自进化通过复用过往经验更新其持久状态,以更高效解决相关任务。现有评估基准存在经济价值任务覆盖不足、训练/测试任务设计难以归因于经验、易受数据污染等问题。本文提出GDPevo,一个基于GDP相关企业工作流的原生进化评估基准,配套全自动化数据生成流水线。其核心机制‘规则混合’将每个企业流程分解为原子业务规则,在训练任务中分布规则子集,并在预留测试任务中重新组合,确保测试性能提升可归因于训练经验。GDPevo涵盖CRM、ERP、金融、医疗、法律及数据密集型工作流,V1版本包含12组共120个任务(每组5个训练+5个测试)。全自动化支持在两天内扩展至240任务(V2),有效应对数据污染。使用GDPevo评估四种含框架与模型的智能体,在四种监督类型下,自进化一致提升未见任务准确率,最高达16.44个百分点。但最优进化智能体仍远低于91.6%的全知理想上限,表明当前自进化能力尚未充分实现。项目代码、基准与结果已公开于https://github.com/Prism-Shadow/GDPevo。
原文摘要 · Abstract (English)
Agent self-evolution updates an agent's persistent state from prior experience and reuses it to solve related tasks more effectively. Evaluating self-evolution is difficult: existing benchmarks provide limited coverage of economically valuable task domains, do not always design training and test tasks such that test-time gains can be attributed to training experience, and remain vulnerable to data contamination. We present GDPevo, an evolution-native benchmark grounded in GDP-related enterprise workflows, together with the fully automated data pipeline that generates it. Its core mechanism, rule hybridization, decomposes each enterprise workflow into atomic business rules, distributes subsets of these rules across training tasks, and recombines them in held-out test tasks so that test-time gains are attributable. GDPevo spans CRM, ERP, finance, healthcare, legal, and data-centric workflows. Its V1 release contains 120 tasks in 12 groups, with five training and five held-out test tasks per group. Full automation enables the pipeline to expand the suite to 240 tasks in 24 groups (V2) within two days, providing a practical response to contamination. Using GDPevo, we evaluate four agents, each comprising a harness and a model, under four supervision types. Self-evolution consistently improves held-out accuracy by up to 16.44 percentage points. But the best evolved agents remain far below the fully informed oracle ceiling of 91.6%, indicating that the self-evolution ability of current agents remains far from fully realized. We publicly release the pipeline, benchmark, and full evaluation results at https://github.com/Prism-Shadow/GDPevo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。