用测试用例做提示，评测大模型写代码能力。
Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation
- 以测试用例为输入生成代码，模拟真实开发流程。
- 19个前沿模型在1000个任务中表现参差，长提示下指令理解力下降。
- 适合关注代码生成实用性和测试驱动开发的研究者。
我们提出WebApp1K，一个面向大语言模型（LLMs）在测试驱动开发（TDD）任务中表现的新基准。该基准中，测试用例既作为生成代码的提示，也作为验证标准，不依赖自然语言描述，更贴近真实软件开发实践。基准包含20个应用领域中的1000个多样化挑战，评估模型在上下文长度限制和多特征复杂性下的代码生成能力。实验表明，指令遵循能力和上下文学习是TDD成功的关键，超越了通用编程能力或预训练知识的重要性。对19个前沿模型的全面评估揭示了性能瓶颈，如长提示中的指令丢失，并进行了涵盖多种根本原因的详细错误分析。本工作凸显了TDD专用基准的实际价值，为提升大模型在严格、应用导向的编码场景中的能力奠定了基础。
原文摘要 · Abstract (English)
We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。