测试通用电脑操作代理能否像专业程序员一样写代码。
Programming with Pixels: Can Computer-Use Agents do Software Engineering?
- 用视觉控制IDE完成多种编程任务，构建首个软件工程专用评测环境。
- 仅靠视觉交互时表现差，但接入文件编辑和命令行接口后性能大幅提升。
- 适合研究通用智能体在复杂任务中的潜力，尤其关注环境利用能力。
计算机使用代理（CUAs）有望执行各类通用任务，但现有评估多集中于简单场景，尚不清楚其是否能自动化更复杂的专业工作如软件工程。为此，我们提出《编程与像素》（PwP），首个面向软件工程的综合性电脑使用环境，代理通过视觉控制IDE完成多样化编程任务。为实现全面评估，我们还构建了包含15项新旧任务的基准测试集PwP-Bench，涵盖多种模态、编程语言与技能层级。我们对前沿开源与闭源CUAs进行了广泛测试，发现仅依赖视觉交互时，其表现显著逊于专业编码代理；但若赋予其文件编辑与bash命令两个基础API，性能跃升，常接近甚至达到专业代理水平，即便模型本身是任务无关设计。进一步提供文本形式的IDE工具接口后，所有模型均获持续提升。分析表明，当前CUAs主要受限于视觉定位能力不足，以及未能充分利用环境资源，仍有巨大改进空间。PwP确立了软件工程作为检验通用代理在高阶任务中是否可达专家水平的自然领域。代码与数据已公开于https://programmingwithpixels.com
原文摘要 · Abstract (English)
Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce $\texttt{Programming with Pixels}$ (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at https://programmingwithpixels.com
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。