测试编码智能体优化真实大模型推理任务的能力，发现其虽能定位瓶颈但难实现有效修复。
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
- 构建真实推理优化任务基准，融合执行与LLM评估指标
- 54个实际优化任务中，智能体普遍能识别瓶颈但难生成可用补丁
- 相同模型不同框架表现差异大，说明代码结构设计比模型本身更重要
我们提出ISO-Bench，一个用于测试编码智能体在真实世界推理优化任务中能力的基准。这些任务源自vLLM和SGLang两大主流大模型服务框架，每项任务提供代码库和性能瓶颈描述，要求智能体生成可验证的优化补丁，并与专家方案对比。我们从合并的Pull Request中筛选出54个具有可测量性能提升的任务。现有基准多依赖运行时指标，易被操纵而无法反映代码修改的真实意图。因此，我们结合执行基础（硬性）与LLM辅助（软性）两类指标，证明二者缺一不可。在评估闭源与开源自编码智能体时发现，无单一智能体在所有代码库上领先。令人意外的是，智能体常能准确识别瓶颈，却难以生成有效解决方案。此外，相同底层模型的不同智能体表现差异显著，表明架构设计与工具链同样关键。
原文摘要 · Abstract (English)
We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。