用竞赛方式评测代码模型如何自主实现开放目标,发现其战略思维和长期维护能力严重不足。
CodeClash: Benchmarking Goal-Oriented Software Engineering
- 模型在多轮对抗中自主改进代码,通过竞争机制评估目标达成效果
- 顶级模型在所有回合中均不敌人类专家,且代码库随时间变得混乱冗余
- 适合关注代码生成智能、人机协作与自主开发的开发者与研究者
当前代码评测基准聚焦于具体明确的任务,如修复特定漏洞或编写测试用例。然而真实软件开发围绕高阶目标展开,如提升用户留存率或降低成本。评估语言模型能否在无明确指导的情况下,持续迭代代码以实现开放性目标,仍是未解挑战。为此,我们提出CodeClash,一个让模型在多轮竞赛中构建最优代码库以达成竞争目标的基准。每轮分两个阶段:模型编辑代码,随后在代码竞技场中对抗,胜负由得分最大化、资源获取或生存等目标决定。模型需自行判断是否写注释、分析文档、查看对手日志或构建测试套件。我们运行1680场竞赛(共25,200轮),评估8个语言模型在6个竞技场的表现。结果表明,尽管模型展现出多样化的开发风格,但在战略推理方面存在根本局限;同时难以长期维护代码库,导致代码逐渐杂乱冗余。最先进模型在所有回合中均败于人类专家。我们开源CodeClash,以推动自主目标导向代码开发的研究。
原文摘要 · Abstract (English)
Current benchmarks for coding evaluate language models (LMs) on concrete, well-specified tasks such as fixing specific bugs or writing targeted tests. However, human programmers do not spend all day incessantly addressing isolated tasks. Instead, real-world software development is grounded in the pursuit of high-level goals, like improving user retention or reducing costs. Evaluating whether LMs can also iteratively develop code to better accomplish open-ended objectives without any explicit guidance remains an open challenge. To address this, we introduce CodeClash, a benchmark where LMs compete in multi-round tournaments to build the best codebase for achieving a competitive objective. Each round proceeds in two phases: agents edit their code, then their codebases compete head-to-head in a code arena that determines winners based on objectives like score maximization, resource acquisition, or survival. Whether it's writing notes, scrutinizing documentation, analyzing competition logs, or creating test suites, models must decide for themselves how to improve their codebases both absolutely and against their opponents. We run 1680 tournaments (25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal that while models exhibit diverse development styles, they share fundamental limitations in strategic reasoning. Models also struggle with long-term codebase maintenance, as repositories become progressively messy and redundant. These limitations are stark: top models lose every round against expert human programmers. We open-source CodeClash to advance the study of autonomous, goal-oriented code development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。