用对话式评测让大模型在复杂编程中边改边学，发现静态测试看不见的问题。
Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks
- 通过问答交互动态评估大模型编程能力，每步只给必要提示。
- 在55个任务上验证，提示相关性与有效性经专家标注确认。
- 适合研究代码生成、人机协作的开发者和评测人员参考。
标准的单轮静态评测难以衡量大语言模型在复杂软件工程任务中的真实能力。本文提出一种新型交互式评测框架，通过结构化反馈对话评估大模型在多需求编程任务上的表现。每个任务建模为需求依赖图，由了解正确解的‘面试官’模型提供最小且精准的提示，帮助‘应聘者’模型修正错误并满足约束条件。该动态流程可揭示模型行为的细粒度诊断信息，发现静态基准无法捕捉的优势与系统性弱点。我们在包含55个精选编程任务的DevAI基准上补充了真实解，并通过专家标注评估提示的相关性与实用性。结果表明，动态评测对推动协作式代码生成智能体的发展至关重要。
原文摘要 · Abstract (English)
Standard single-turn, static benchmarks fall short in evaluating the nuanced capabilities of Large Language Models (LLMs) on complex tasks such as software engineering. In this work, we propose a novel interactive evaluation framework that assesses LLMs on multi-requirement programming tasks through structured, feedback-driven dialogue. Each task is modeled as a requirement dependency graph, and an ``interviewer'' LLM, aware of the ground-truth solution, provides minimal, targeted hints to an ``interviewee'' model to help correct errors and fulfill target constraints. This dynamic protocol enables fine-grained diagnostic insights into model behavior, uncovering strengths and systematic weaknesses that static benchmarks fail to measure. We build on DevAI, a benchmark of 55 curated programming tasks, by adding ground-truth solutions and evaluating the relevance and utility of interviewer hints through expert annotation. Our results highlight the importance of dynamic evaluation in advancing the development of collaborative code-generating agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。