首个评估代码迭代生成能力的基准，推动大模型在真实开发中复用代码。
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
- 通过多轮对话复用已有函数实现新功能，模拟真实开发流程。
- 5000+竞赛题与GitHub项目构建双数据集，覆盖复杂依赖场景。
- 发现模型表现随依赖复杂度上升而下降，揭示关键挑战。
现代软件开发要求代码具备可维护性、可测试性和可扩展性，通过模块化组件和对现有代码的迭代复用来实现。本文将这一多轮、迭代式开发范式形式化为「代码流（codeflow）」，并提出首个专门评估大语言模型代码流能力的基准——CodeFlowBench。该基准包含两个互补部分：CodeFlowBench-Comp，基于自动化流水线更新的5000+个来自Codeforces的编程竞赛题目；以及CodeFlowBench-Repo，源自GitHub仓库的真实项目，更贴近实际开发场景。此外，引入新型评估框架，包含双评估协议和基于依赖树的结构化指标。大量实验表明，在多轮代码流任务中模型性能显著下降；深入分析显示，模型表现与依赖复杂度呈反比。这些发现不仅揭示了支持真实开发工作流的关键挑战，也确立了CodeFlowBench作为推进代码生成研究的重要工具。
原文摘要 · Abstract (English)
Modern software development demands code that is maintainable, testable, and scalable by organizing the implementation into modular components with iterative reuse of existing codes. We formalize this iterative, multi-turn paradigm as codeflow and introduce CodeFlowBench, the first benchmark designed to comprehensively evaluate LLMs' ability to perform codeflow - implementing new functionality by reusing existing functions over multiple turns. CodeFlowBench comprises two complementary components: CodeFlowBench-Comp, a core collection of 5,000+ competitive programming problems from Codeforces updated via an automated pipeline and CodeFlowBench-Repo, which is sourced from GitHub repositories to better reflect real-world scenarios. Furthermore, a novel evaluation framework featured dual assessment protocol and structural metrics derived from dependency trees is introduced. Extensive experiments reveal significant performance degradation in multi-turn codeflow scenarios. Furthermore, our in-depth analysis illustrates that model performance inversely correlates with dependency complexity. These findings not only highlight the critical challenges for supporting real-world workflows, but also establish CodeFlowBench as an essential tool for advancing code generation research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。