构建真实开发场景的代码生成评测集，揭示当前模型实际表现短板。
DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
- 基于真实开发者行为数据生成1800个任务，覆盖六语言六类别。
- 顶尖模型仅43.5%通过率，暴露语法、语义与实用性的多重缺陷。
- 适合关注模型落地效果的研究者与工程团队使用。
DevBench 是一个基于真实开发者行为数据的代码生成评估基准，涵盖6种编程语言和6类任务，共1800个评估实例。数据通过多来源生成器模型合成，以减少单一数据源偏差。该基准强调生态有效性，避免训练数据污染，并支持细粒度诊断。评估结合功能正确性、相似性指标及大模型判别，重点关注实用性与上下文相关性。对9个主流模型的测试显示，最强模型的Pass@1仅为43.5%，证实其挑战性，并揭示了模型在语法精度、语义推理和实际可用性方面的差异。该基准为模型选型与改进提供可操作洞察，弥补了现有评测中实践信息缺失的问题。
原文摘要 · Abstract (English)
DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry and synthesized using generator models from multiple provider families to mitigate single-source bias. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, with the strongest achieving only 43.5% Pass@1, confirming the benchmark remains challenging and revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement, detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。