arXiv:2509.24091cs.SEcs.AI2025-09被引 5

评测智能体修复真实性能缺陷的能力,发现现有工具效果极差。

PerfBench: Can Agents Resolve Real-World Performance Bugs?

  • 构建81个真实.NET项目中的性能缺陷任务集
  • 新基准允许智能体自动生成性能测试并验证修复结果
  • 当前最先进智能体成功率仅3%,改进后达20%

性能缺陷是不导致功能失败但浪费计算资源的软件问题,检测和修复极为困难。现有软件工程智能体评估多聚焦功能正确性,忽视非功能性问题如性能缺陷。本文提出PerfBench,一个包含81个来自GitHub热门.NET仓库的真实性能缺陷修复任务的基准。不同于依赖预设测试套件的现有方法,PerfBench采用新型评估机制,使智能体可自主生成性能基准,并通过比较开发者修复与智能体修复的执行指标来验证结果。每个任务均源自实际开发者提交的性能相关修复,经人工专家验证,确保真实有效性。评估显示,当前最先进的OpenHands智能体在该基准上成功率仅为约3%。我们开发了OpenHands-Perf-Agent,引入性能感知工具与指令,将成功率提升至约20%。结果表明,提供恰当的基准生成指导和输出处理工具可显著提升智能体表现,但仍存改进空间。PerfBench为推动智能体在性能修复方面的能力提供了挑战性测试集。

原文摘要 · Abstract (English)

Performance bugs are inefficiencies in software that waste computational resources without causing functional failures, making them particularly challenging to detect and fix. While recent advances in Software Engineering agents have shown promise in automated bug fixing, existing benchmarks primarily focus on functional correctness and fail to evaluate agents' abilities to identify and resolve non-functional issues like performance bugs. We introduce PerfBench, a benchmark comprising 81 real-world performance bug-fixing tasks from popular .NET repositories on GitHub. Unlike existing benchmarks that rely on pre-existing test suites, PerfBench features a novel evaluation harness that allows agents to generate their own performance benchmarks and validates fixes by comparing execution metrics collected for developer fix and agent fix. Each task in PerfBench is derived from actual developer fixes linked to performance-related issues, which are then verified by human experts, ensuring real-world relevance. Our evaluation reveals that current state-of-the-art coding agents struggle with performance optimization tasks, with baseline OpenHands agent achieving only a ~3% success rate on our benchmark. We develop OpenHands-Perf-Agent, which incorporates performance-aware tooling and instructions and achieves a ~20% success rate on the benchmark. We show that by ensuring the agent has proper instructions to benchmark its changes and tooling for benchmark output processing, we can improve the agent performance significantly, but room for improvement still remains. PerfBench provides a challenging test set for furthering the capabilities of agents in fixing performance issues.

智能体性能优化软件工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。