用约束优化统一生成任务，解决智能体评估中任务不一致问题。
Anchor: Mitigating Artifact Drift in Agent Benchmark Generation
- 将业务流程转化为约束优化问题，统一生成指令、环境与验证规则。
- 生成300个长周期任务，模型仅17.4%能完全达标，体现真实挑战性。
- 适合研究可验证评估环境的AI系统开发者与企业级智能体研究人员。
AI智能体开始承担有价值的长周期企业任务，但训练与评估环境仍难以兼顾真实性、可验证性和规模。当前任务生成常因指令、环境、评判标准和验证器由松散流程生成而出现不一致，导致任务不可解、奖励可操纵或逻辑矛盾。本文提出Anchor，一种将领域专家定义的业务流程形式化为约束优化程序的任务生成管道。从单一参数化规范出发，该管道联合生成自然语言指令、环境配置、求解器认证的真值解及基于状态的验证器。调整参数可生成难度可控、最优解已知的新任务，构建出与测试框架无关、奖励仅依赖最终业务正确性的环境。我们应用Anchor构建ERP-Bench：一个涵盖生产级ERP系统中采购与制造流程的300个长周期任务基准。结果表明，生成参数可预测任务实际难度；前沿模型在26.1%的试验中满足显式约束，但仅17.4%达到完全最优解。整体上，Anchor与ERP-Bench为构建经济价值导向的可审计智能体评估环境提供了可行方案。代码与数据集已在erpbench.ai公开。
原文摘要 · Abstract (English)
AI agents are beginning to complete valuable, long-horizon business operations tasks, but training and evaluation environments for enterprise work still struggle to balance realism, verifiability, and scale. Environment and task creation frequently suffers from a failure mode we call artifact drift: when instructions, environments, oracles, and verifiers are created by loosely coupled processes, they frequently disagree on what a task requires, producing environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a task-generation pipeline that formalizes domain experts' specifications of business workflows into constraint optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With Anchor, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. We find that generation parameters predict realized difficulty, and that frontier models satisfy explicit task constraints in 26.1% of trials but reach a fully optimal solution in only 17.4% of trials. Overall, we show that Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments for economically valuable agent work. We release the task generator and ERP-Bench dataset at erpbench.ai
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。