构建真实工程优化基准,测试智能体迭代改进设计能力
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

- 用生成式优化循环评估智能体,持续反馈+约束条件
- 47个工业级任务,模型在固定预算下仍难突破瓶颈
- 深度比宽度更关键,改进频率与幅度均呈幂律衰减
现有大模型智能体评测多聚焦于代码生成或搜索问答等二元成败任务,忽视了真实工程中通过迭代优化可行设计所体现的价值。为此,我们提出Frontier-Eng,一个经人工验证的生成式优化基准,涵盖5大工程领域共47项任务。该基准基于工业级仿真器与验证器,支持可执行反馈和连续奖励信号,并施加硬性可行性约束与固定交互预算。我们使用代表性搜索框架评估8个前沿语言模型,发现即使GPT 5.4表现最稳健,所有模型仍面临挑战。分析表明,改进频率(~1/迭代)和幅度(~1/改进次数)均呈现双幂律衰减。进一步发现:虽宽度提升并行性与多样性,但深度对在固定预算下实现关键突破仍至关重要。Frontier-Eng确立了评估智能体融合领域知识与可执行反馈解决复杂开放工程问题的新标准。
原文摘要 · Abstract (English)
Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。