无需运行代码即可检测编程智能体的运行时错误，提升代码可靠性。
REDO: Execution-Free Runtime Error Detection for COding Agents
- 用大模型结合静态分析，不执行代码就发现运行时错误。
- 在复杂项目中检测错误，准确率比现有方法高11.0%。
- 适合提升编程智能体的鲁棒性，尤其对依赖外部库的任务有用。
随着基于大模型的智能体在解决复杂问题上展现出卓越能力，针对日益复杂的任务开发编程智能体成为研究热点。尽管表现优异，这些编程智能体生成的程序或修改常包含运行时错误，导致代码失败，且难以被静态分析工具检测。增强编程智能体静态识别此类错误的能力可显著提升其整体性能。本文提出执行无关的运行时错误检测方法（REDO），通过将大模型与静态分析工具结合，在不执行代码的前提下检测编程智能体产生的运行时错误。此外，我们基于SWE-Bench（lite）构建了一个基准任务SWE-Bench-Error-Detection（SWEDE），用于评估具有复杂外部依赖的仓库级问题中的错误检测能力。通过多维度定量与定性分析，实验表明RED O在各项错误检测任务中优于当前最先进方法，准确率提升11.0%，加权F1得分提高9.1%；同时揭示了引入大模型在错误检测中的优势。
原文摘要 · Abstract (English)
As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for COding Agents (REDO), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection (SWEDE), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score; and provide insights into the advantages of incorporating LLMs for error detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。