评测编程智能体在交互式项目构建中的综合能力。
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
- 从模糊需求出发，模拟真实开发中动态协作场景。
- 基于真实开源项目生成任务，确保可复现与高质量评估。
- 多维度指标评估代码功能、结构与交互质量，适合研究交互式编程代理。
近年来，'氛围化编码'工作流正在改变对编程智能体的期待。它们不再只是完成明确指令的代码补全，而是需通过规划、需求澄清、工具使用、调试和仓库级构建等能力，将不完整的产品意图转化为可运行的软件。然而现有基准仍停留在静态、完全指定的任务上，未能跟上这一变化。本文提出 ICAE-Bench，一个面向交互式项目构建场景的编程智能体评估基准。核心思路是从模糊的产品需求出发，通过自动化用户代理（User Agent）模拟动态开发过程。为保证真实性和可评估性，ICAEBench引入三项关键设计：第一，每个任务均源自具有可执行行为的真实开源仓库，避免无约束模糊需求带来的歧义；第二，通过用户代理数据实现交互的可再现性，使用户能揭示隐藏约束，而不引入新需求或泄露实现细节；第三，采用标准化黑盒测试结合多维诊断，包括功能正确性、语义与API相似度、结构保真度、设计质量和交互质量，以公平评估开放式仓库构建任务。
原文摘要 · Abstract (English)
The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。