构建可复现的对话式代码生成评测环境，精准模拟真实交互反馈。
ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments
- 设计仿真9种交互场景的可复现环境，融合编译、运行和专家级语言反馈。
- 提出静态版基准测试，保留与动态版0.82~0.99的相关性，节省计算成本。
- 揭示反馈类型影响模型表现，弱模型在优质反馈下可超越强模型单轮表现。
大型语言模型（LLMs）在代码生成中表现优异，尤其在交互场景中。然而，现有基准未能涵盖多轮交互中的多样化反馈，制约了对这类场景下模型性能的评估。为此，我们提出了全新的基准体系：首先，构建了CONVCODEWORLD，一个可复现的交互式代码生成评测环境，模拟9种不同交互场景，并系统组合三类反馈——（a）编译反馈；（b）具有不同测试覆盖率的执行反馈；（c）由GPT-4o生成的具备不同专业水平的自然语言反馈。其次，提出CONVCODEBENCH，一种快速静态版本，使用预生成反馈日志，无需动态生成昂贵的语言反馈，仍保持与CONVCODEWORLD高达0.82至0.99的斯皮尔曼等级相关性。第三，对闭源与开源模型（包括R1-Distill）在CONVCODEWORLD上的大量评估揭示关键发现：（a）模型表现显著受反馈质量影响；（b）弱模型在充分反馈下可超越顶尖模型无反馈时的单轮表现；（c）在特定反馈组合上训练会削弱模型对未见反馈组合的适应能力；（d）少轮解决更多问题（高MRR）的模型未必整体解决问题更多（高召回），反之亦然。所有代码与基准数据将公开于https://huggingface.co/spaces/ConvCodeWorld/ConvCodeWorld。
原文摘要 · Abstract (English)
Large language models (LLMs) have proven invaluable for code generation, particularly in interactive settings. However, existing code generation benchmarks fail to capture the diverse feedback encountered in multi-turn interactions, limiting our ability to evaluate LLMs in these contexts. To address this gap, we present a set of novel benchmarks that explicitly model the quality of feedback provided to code generation LLMs. Our contributions are threefold: First, we introduce CONVCODEWORLD, a novel and reproducible environment for benchmarking interactive code generation. CONVCODEWORLD simulates 9 distinct interactive code generation scenarios while systematically combining three types of feedback: (a) compilation feedback; (b) execution feedback with varying test coverage; (c) verbal feedback generated by GPT-4o with different levels of expertise. Second, we introduce CONVCODEBENCH, a fast, static version of benchmark that uses pre-generated feedback logs, eliminating the need for costly dynamic verbal feedback generation while maintaining strong Spearman's rank correlations (0.82 to 0.99) with CONVCODEWORLD. Third, extensive evaluations of both closed-source and open-source LLMs including R1-Distill on CONVCODEWORLD reveal key insights: (a) LLM performance varies significantly based on the feedback provided; (b) Weaker LLMs, with sufficient feedback, can outperform single-turn results of state-of-the-art LLMs without feedback; (c) Training on a specific feedback combination can limit an LLM's ability to utilize unseen combinations; (d) LLMs solve problems in fewer turns (high MRR) may not solve as many problems overall (high Recall), and vice versa. All implementations and benchmarks will be made publicly available at https://huggingface.co/spaces/ConvCodeWorld/ConvCodeWorld
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。