16个前沿大模型在网页代码生成测试中表现差异源于错误频率，而非知识差异。
Insights from Benchmarking Frontier Language Models on Web App Code Generation
- 通过分析代码行和失败分布，发现正确编码比错误编码更复杂。
- 所有模型知识相近，但错误频率差异显著，影响实际可用性。
- 提示工程对降错效果有限，模型可靠性才是关键突破点。
本文通过对16个前沿大语言模型在WebApp1K基准上的评估，揭示了其在生成网页应用代码方面的能力。尽管各模型具备相似的基础知识，但性能差异主要体现在错误发生的频率上。通过分析代码行数（LOC）和失败分布，研究发现编写正确代码比生成错误代码更具挑战性。此外，提示工程在减少错误方面的效果有限，仅在特定情况下有效。这些发现表明，未来编程类大模型的发展应更加注重模型的可靠性与错误最小化。
原文摘要 · Abstract (English)
This paper presents insights from evaluating 16 frontier large language models (LLMs) on the WebApp1K benchmark, a test suite designed to assess the ability of LLMs to generate web application code. The results reveal that while all models possess similar underlying knowledge, their performance is differentiated by the frequency of mistakes they make. By analyzing lines of code (LOC) and failure distributions, we find that writing correct code is more complex than generating incorrect code. Furthermore, prompt engineering shows limited efficacy in reducing errors beyond specific cases. These findings suggest that further advancements in coding LLM should emphasize on model reliability and mistake minimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。