新基准评估代码生成模型对开发者偏好调整的响应能力
CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments
- 构建多语言基准，测试模型对初始约束和后续修改指令的遵循能力
- 在LiveBench任务上发现模型在不同语言和修改目标下表现差异显著
- 适合关注代码生成实用性与指令理解能力的研究者使用
随着大语言模型生成代码能力提升，其性能评估仍面临复杂挑战。现有基准多聚焦功能正确性，忽视真实开发中任务多样性和开发者预期。为此，我们提出一个支持多语言的可扩展基准，用于评估大模型在两类关键场景下的指令遵循能力：遵守初始问题中的预定义约束，以及根据后续指令进行代码优化的能力。本文基于LiveBench编程任务（含自动从Python转译的Java和JavaScript版本）验证了该评估流程。自动化测试结果表明，模型在指令遵循的多个维度上表现不一，揭示了其在跨语言与生成目标上的优势与局限。该基准为代码生成模型提供更全面的评估视角。
原文摘要 · Abstract (English)
As large language models become increasingly capable of generating code, evaluating their performance remains a complex and evolving challenge. Existing benchmarks primarily focus on functional correctness, overlooking the diversity of real-world coding tasks and developer expectations. To this end, we introduce a multi-language benchmark that evaluates LLM instruction-following capabilities and is extensible to operate on any set of standalone coding problems. Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions. For this paper's analysis, we empirically evaluated our benchmarking pipeline with programming tasks from LiveBench, that are also automatically translated from Python into Java and JavaScript. Our automated benchmark reveals that models exhibit differing levels of performance across multiple dimensions of instruction-following. Our benchmarking pipeline provides a more comprehensive evaluation of code generation models, highlighting their strengths and limitations across languages and generation goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。