新基准Web-Bench模拟真实网页开发流程，挑战大模型全流程编码能力。
Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks
- 基于网页标准与框架设计50个需逐项完成的项目任务
- 顶尖模型Claude 3.7 Sonnet在该基准上仅达25.1%通过率
- 适合评估大模型在真实工程场景中的自主开发能力
大型语言模型（LLMs）在编程领域的应用迅速发展，从代码助手到自主编码代理，再到通过自然语言生成完整项目。早期的代码基准主要关注代码生成准确率，但这些基准已趋于饱和——例如HumanEval Pass@1已达99.4%，MBPP达94.2%。为应对基准饱和问题，现有基于软件工程的方法虽有成效，但其自身也面临饱和风险。为此，我们提出新基准Web-Bench，包含50个完整项目，每项目含20个具有依赖关系的任务，按顺序实现功能，模拟真实开发者工作流。设计时覆盖网页开发的核心要素：网页标准与主流框架。所有项目由拥有5至10年经验的工程师设计，单个项目平均耗时4至8小时，难度高。在所提基准代理（Web-Agent）上，当前最优模型Claude 3.7 Sonnet仅获25.1% Pass@1，显著低于SWE-Bench的Verified（65.4%）与Full（33.8%）得分。最后指出，各开发领域中标准与框架分别代表基础认知与效率工具，大模型需针对二者进行针对性优化。
原文摘要 · Abstract (English)
The application of large language models (LLMs) in the field of coding is evolving rapidly: from code assistants, to autonomous coding agents, and then to generating complete projects through natural language. Early LLM code benchmarks primarily focused on code generation accuracy, but these benchmarks have gradually become saturated. Benchmark saturation weakens their guiding role for LLMs. For example, HumanEval Pass@1 has reached 99.4% and MBPP 94.2%. Among various attempts to address benchmark saturation, approaches based on software engineering have stood out, but the saturation of existing software engineering benchmarks is rapidly increasing. To address this, we propose a new benchmark, Web-Bench, which contains 50 projects, each consisting of 20 tasks with sequential dependencies. The tasks implement project features in sequence, simulating real-world human development workflows. When designing Web-Bench, we aim to cover the foundational elements of Web development: Web Standards and Web Frameworks. Given the scale and complexity of these projects, which were designed by engineers with 5 to 10 years of experience, each presents a significant challenge. On average, a single project takes 4 to 8 hours for a senior engineer to complete. On our given benchmark agent (Web-Agent), SOTA (Claude 3.7 Sonnet) achieves only 25.1% Pass@1, significantly lower (better) than SWE-Bench's Verified (65.4%) and Full (33.8%) scores. Finally, we discuss that in any development field, Standards and Frameworks represent foundational knowledge and efficiency tools, respectively, and LLMs require optimization tailored to them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。